[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:23:21 dist-test-slave-rnrw krb5kdc[12607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:23:21 dist-test-slave-rnrw krb5kdc[12607](info): set up 2 sockets
Aug 09 20:23:21 dist-test-slave-rnrw krb5kdc[12607](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 20:23:23.560847 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.664s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:23:23 dist-test-slave-rnrw krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:23:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:23:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:23:23.750065 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:35061
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:39251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:24.267448 12622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:24.267834 12622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:24.268015 12622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:24.277279 12622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:23:24.277477 12622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:24.277576 12622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:24.277721 12622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:23:24.277801 12622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:23:24.289781 12622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:39251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:35061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:24.291153 12622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:24.292649 12622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:24.309309 12630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:24.309640 12628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:24.310076 12627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:24.311012 12622 server_base.cc:1034] running on GCE node
I20260809 20:23:24.311982 12622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:24.313787 12622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:24.315070 12622 hybrid_clock.cc:648] HybridClock initialized: now 1786307004314985 us; error 59 us; skew 500 ppm
I20260809 20:23:24.315505 12622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:24 dist-test-slave-rnrw krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307004, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:24.319742 12622 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:23:24.321712 12622 webserver.cc:460] Webserver started at http://127.12.78.62:44795/ using document root <none> and password file <none>
I20260809 20:23:24.322592 12622 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:24.322850 12622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:24.323262 12622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:24.326825 12622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "56fa0c07f34a48d582421f1ae9f70952"
format_stamp: "Formatted at 2026-08-09 20:23:24 on dist-test-slave-rnrw"
server_key: "4821e7d6d7c12189d9075bfe8a4ab950"
server_key_iv: "75c069041ab372d2a53dfeee550dc6ef"
server_key_version: "encryptionkey@0"
I20260809 20:23:24.327690 12622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "56fa0c07f34a48d582421f1ae9f70952"
format_stamp: "Formatted at 2026-08-09 20:23:24 on dist-test-slave-rnrw"
server_key: "4821e7d6d7c12189d9075bfe8a4ab950"
server_key_iv: "75c069041ab372d2a53dfeee550dc6ef"
server_key_version: "encryptionkey@0"
I20260809 20:23:24.333645 12622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260809 20:23:24.338243 12638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:24.339668 12622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:23:24.339867 12622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "56fa0c07f34a48d582421f1ae9f70952"
format_stamp: "Formatted at 2026-08-09 20:23:24 on dist-test-slave-rnrw"
server_key: "4821e7d6d7c12189d9075bfe8a4ab950"
server_key_iv: "75c069041ab372d2a53dfeee550dc6ef"
server_key_version: "encryptionkey@0"
I20260809 20:23:24.340086 12622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:24.358577 12622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:24.364041 12622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:24.364507 12622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:24.397094 12622 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:35061
I20260809 20:23:24.397287 12691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:35061 every 8 connection(s)
I20260809 20:23:24.399067 12622 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:23:24.402937 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 12622
I20260809 20:23:24.403403 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:23:24.404505 12692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:24.406121 12600 external_mini_cluster.cc:1428] Setting key 620bcdfcfdeb0ba3f32d71d4a060937a
I20260809 20:23:24.423918 12692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952: Bootstrap starting.
I20260809 20:23:24.429056 12692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:24.430765 12692 log.cc:824] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:24.435987 12692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952: No bootstrap required, opened a new log
I20260809 20:23:24.450595 12692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56fa0c07f34a48d582421f1ae9f70952" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35061 } }
I20260809 20:23:24.451416 12692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:24.451553 12692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56fa0c07f34a48d582421f1ae9f70952, State: Initialized, Role: FOLLOWER
I20260809 20:23:24.452610 12692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56fa0c07f34a48d582421f1ae9f70952" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35061 } }
Aug 09 20:23:24 dist-test-slave-rnrw krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:24.452911 12692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:23:24.453119 12692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:23:24.453382 12692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:24.456535 12692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56fa0c07f34a48d582421f1ae9f70952" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35061 } }
I20260809 20:23:24.457361 12692 leader_election.cc:302] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56fa0c07f34a48d582421f1ae9f70952; no voters:
I20260809 20:23:24.458437 12692 leader_election.cc:288] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:23:24.458580 12697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:24.460956 12697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [term 1 LEADER]: Becoming Leader. State: Replica: 56fa0c07f34a48d582421f1ae9f70952, State: Running, Role: LEADER
I20260809 20:23:24.461486 12692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:23:24.461611 12697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56fa0c07f34a48d582421f1ae9f70952" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35061 } }
I20260809 20:23:24.468488 12698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56fa0c07f34a48d582421f1ae9f70952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56fa0c07f34a48d582421f1ae9f70952" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35061 } } }
I20260809 20:23:24.468036 12699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56fa0c07f34a48d582421f1ae9f70952. Latest consensus state: current_term: 1 leader_uuid: "56fa0c07f34a48d582421f1ae9f70952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56fa0c07f34a48d582421f1ae9f70952" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35061 } } }
I20260809 20:23:24.469722 12698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:24.469817 12699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952 [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:24.472774 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:24.412953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.12.78.62:35061)
0809 20:23:24.414350 (+ 1397us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:24.414366 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:24.416418 (+ 2052us) server_negotiation.cc:407] Connection header received
0809 20:23:24.432837 (+ 16419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:24.432887 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:24.433690 (+ 803us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:24.434128 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:24.439253 (+ 5125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:24.441111 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:24.444216 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:24.445016 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:24.445316 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:24.454652 (+ 9336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:24.454738 (+ 86us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:24.454774 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:24.454818 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:24.460018 (+ 5200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:24.461629 (+ 1611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:24.461639 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:24.461647 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:24.461879 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:24.462489 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:24.462494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:24.462497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:24.463551 (+ 1054us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:24.466854 (+ 3303us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:24.467298 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 20:23:24.467872 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":583,"threads_started":1}
I20260809 20:23:24.483335 12705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:23:24.490903 12705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:23:24.503098 12705 catalog_manager.cc:1347] Generated new cluster ID: d586199d1723480e91a2e5dba5670f7f
I20260809 20:23:24.503278 12705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:23:24.517890 12705 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:23:24.520345 12705 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:23:24.533699 12705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56fa0c07f34a48d582421f1ae9f70952: Generated new TSK 0
I20260809 20:23:24.534981 12705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:23:24.588515 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:35061
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:39251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:25.090762 12718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:25.091097 12718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:25.091367 12718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:25.100204 12718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:25.100404 12718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:25.100670 12718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:23:25.112210 12718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:39251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:35061
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:25.113629 12718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:25.115056 12718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:25.130728 12724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:25.130746 12726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:25.130993 12723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:25.132411 12718 server_base.cc:1034] running on GCE node
I20260809 20:23:25.133145 12718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:25.134171 12718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:25.135526 12718 hybrid_clock.cc:648] HybridClock initialized: now 1786307005135416 us; error 35 us; skew 500 ppm
I20260809 20:23:25.135910 12718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:25 dist-test-slave-rnrw krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307005, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:25.139863 12718 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:23:25.141469 12718 webserver.cc:460] Webserver started at http://127.12.78.1:36969/ using document root <none> and password file <none>
I20260809 20:23:25.142186 12718 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:25.142305 12718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:25.142661 12718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:25.145889 12718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "515580fbdb6543239dcbb951febf2034"
format_stamp: "Formatted at 2026-08-09 20:23:25 on dist-test-slave-rnrw"
server_key: "c1e2da7b71ac925dc37529d0159ab681"
server_key_iv: "4f0dd28f005fa8d8c412f36b0db5937f"
server_key_version: "encryptionkey@0"
I20260809 20:23:25.146705 12718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "515580fbdb6543239dcbb951febf2034"
format_stamp: "Formatted at 2026-08-09 20:23:25 on dist-test-slave-rnrw"
server_key: "c1e2da7b71ac925dc37529d0159ab681"
server_key_iv: "4f0dd28f005fa8d8c412f36b0db5937f"
server_key_version: "encryptionkey@0"
I20260809 20:23:25.152417 12718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 20:23:25.156425 12734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:25.157927 12718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:23:25.158169 12718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "515580fbdb6543239dcbb951febf2034"
format_stamp: "Formatted at 2026-08-09 20:23:25 on dist-test-slave-rnrw"
server_key: "c1e2da7b71ac925dc37529d0159ab681"
server_key_iv: "4f0dd28f005fa8d8c412f36b0db5937f"
server_key_version: "encryptionkey@0"
I20260809 20:23:25.158434 12718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:25.173893 12718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:25.178766 12718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:25.179111 12718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:25.180437 12718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:25.182440 12718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:25.182550 12718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:25.182783 12718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:25.182938 12718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:25.245172 12718 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:35183
I20260809 20:23:25.245208 12852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:35183 every 8 connection(s)
I20260809 20:23:25.246673 12718 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:23:25.254896 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 12718
I20260809 20:23:25.255193 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:23:25.256166 12600 external_mini_cluster.cc:1428] Setting key ebc8f0515b86b877e95f03fa3fb09cab
Aug 09 20:23:25 dist-test-slave-rnrw krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307005, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:25.275843 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:25.249912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:48561 (local address 127.12.78.62:35061)
0809 20:23:25.250470 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:25.250475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:25.251770 (+ 1295us) server_negotiation.cc:407] Connection header received
0809 20:23:25.254478 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:25.254483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:25.254553 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:25.254655 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:25.257743 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:25.259099 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:25.262323 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:25.263161 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:25.263293 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:25.268047 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:25.268053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:25.268056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:25.268093 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:25.271380 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:25.272453 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:25.272462 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:25.272468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:25.272669 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:25.273216 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:25.273224 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:25.273228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:25.273690 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:25.273892 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:25.274715 (+ 823us) server_negotiation.cc:299] Negotiation successful
0809 20:23:25.275108 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":311,"threads_started":1}
I20260809 20:23:25.276401 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:25.250556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:35061 (local address 127.12.78.1:48561)
0809 20:23:25.251598 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:25.251636 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:25.254148 (+ 2512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:25.255085 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:25.255097 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:25.256274 (+ 1177us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:25.257487 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:25.257516 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:25.259251 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:25.259257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:25.262130 (+ 2873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:25.262150 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:25.263475 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:25.263485 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:25.263869 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:25.264864 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:25.264907 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:25.267764 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:25.271552 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:25.271562 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:25.271581 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:25.272243 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:25.272787 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:25.272791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:25.272793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:25.273056 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:25.273838 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:25.273844 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:25.274243 (+ 399us) client_negotiation.cc:769] Sending connection context
0809 20:23:25.274531 (+ 288us) client_negotiation.cc:240] Negotiation successful
0809 20:23:25.274977 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":602,"threads_started":1}
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
I20260809 20:23:25.278482 12853 heartbeater.cc:347] Connected to a master server at 127.12.78.62:35061
I20260809 20:23:25.278846 12853 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:25.279896 12853 heartbeater.cc:511] Master 127.12.78.62:35061 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
I20260809 20:23:25.282428 12656 ts_manager.cc:194] Registered new tserver with Master: 515580fbdb6543239dcbb951febf2034 (127.12.78.1:35183)
I20260809 20:23:25.284075 12656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:48561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:23:25.307370 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:35061
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:39251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:25.805456 12859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:25.805734 12859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:25.805869 12859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:25.814604 12859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:25.814783 12859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:25.814915 12859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:23:25.826061 12859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:39251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:35061
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:25.827458 12859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:25.828904 12859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:25.844738 12864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:25.845161 12866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:25.845603 12868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:25.850404 12859 server_base.cc:1034] running on GCE node
I20260809 20:23:25.851200 12859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:25.852391 12859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:25.853638 12859 hybrid_clock.cc:648] HybridClock initialized: now 1786307005853568 us; error 59 us; skew 500 ppm
I20260809 20:23:25.853984 12859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:25 dist-test-slave-rnrw krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307005, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:25.858155 12859 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:23:25.860018 12859 webserver.cc:460] Webserver started at http://127.12.78.2:43277/ using document root <none> and password file <none>
I20260809 20:23:25.860903 12859 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:25.861052 12859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:25.861374 12859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:25.865064 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "886277bc98994c2f9c9ae99cc59c6055"
format_stamp: "Formatted at 2026-08-09 20:23:25 on dist-test-slave-rnrw"
server_key: "461818c207335b64a28620ea9b005651"
server_key_iv: "84c9f8eeaf3cf2068fe2bd585189f0f9"
server_key_version: "encryptionkey@0"
I20260809 20:23:25.865921 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "886277bc98994c2f9c9ae99cc59c6055"
format_stamp: "Formatted at 2026-08-09 20:23:25 on dist-test-slave-rnrw"
server_key: "461818c207335b64a28620ea9b005651"
server_key_iv: "84c9f8eeaf3cf2068fe2bd585189f0f9"
server_key_version: "encryptionkey@0"
I20260809 20:23:25.871874 12859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:23:25.876160 12875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:25.877823 12859 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 20:23:25.878059 12859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "886277bc98994c2f9c9ae99cc59c6055"
format_stamp: "Formatted at 2026-08-09 20:23:25 on dist-test-slave-rnrw"
server_key: "461818c207335b64a28620ea9b005651"
server_key_iv: "84c9f8eeaf3cf2068fe2bd585189f0f9"
server_key_version: "encryptionkey@0"
I20260809 20:23:25.878346 12859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:25.897449 12859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:25.902072 12859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:25.902379 12859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:25.903635 12859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:25.905396 12859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:25.905505 12859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:25.905664 12859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:25.905776 12859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:25.964627 12859 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:32943
I20260809 20:23:25.964677 12991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:32943 every 8 connection(s)
I20260809 20:23:25.966105 12859 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:23:25.971771 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 12859
I20260809 20:23:25.972020 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:23:25.972738 12600 external_mini_cluster.cc:1428] Setting key 6c3232e82d19714e88ac0ac0b12a7c7b
Aug 09 20:23:25 dist-test-slave-rnrw krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307005, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
I20260809 20:23:25.995149 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:25.969351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:48021 (local address 127.12.78.62:35061)
0809 20:23:25.969873 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:25.969880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:25.971123 (+ 1243us) server_negotiation.cc:407] Connection header received
0809 20:23:25.974237 (+ 3114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:25.974242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:25.974322 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:25.974440 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:25.977066 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:25.978028 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:25.980919 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:25.981523 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:25.981667 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:25.986134 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:25.986143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:25.986151 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:25.986203 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:25.990756 (+ 4553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:25.991802 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:25.991810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:25.991814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:25.992108 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:25.992753 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:25.992760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:25.992764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:25.993194 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:25.993367 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:25.994271 (+ 904us) server_negotiation.cc:299] Negotiation successful
0809 20:23:25.994595 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":241,"threads_started":1}
I20260809 20:23:25.995915 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:25.970032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:35061 (local address 127.12.78.2:48021)
0809 20:23:25.970952 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:25.970989 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:25.973932 (+ 2943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:25.974698 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:25.974707 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:25.975817 (+ 1110us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:25.976856 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:25.976892 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:25.978273 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:25.978282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:25.980734 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:25.980754 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:25.981760 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:25.981765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:25.982025 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:25.982791 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:25.982838 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:25.985846 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:25.990974 (+ 5128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:25.990982 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:25.990998 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:25.991601 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:25.992309 (+ 708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:25.992313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:25.992316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:25.992602 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:25.993387 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:25.993398 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:25.993807 (+ 409us) client_negotiation.cc:769] Sending connection context
0809 20:23:25.994087 (+ 280us) client_negotiation.cc:240] Negotiation successful
0809 20:23:25.994537 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":493,"threads_started":1}
I20260809 20:23:25.997768 12992 heartbeater.cc:347] Connected to a master server at 127.12.78.62:35061
I20260809 20:23:25.998216 12992 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
I20260809 20:23:25.999399 12992 heartbeater.cc:511] Master 127.12.78.62:35061 requested a full tablet report, sending...
I20260809 20:23:26.001446 12656 ts_manager.cc:194] Registered new tserver with Master: 886277bc98994c2f9c9ae99cc59c6055 (127.12.78.2:32943)
I20260809 20:23:26.002319 12656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:48021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:23:26.024458 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:35061
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:39251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:23:26.287910 12853 heartbeater.cc:503] Master 127.12.78.62:35061 was elected leader, sending a full tablet report...
W20260809 20:23:26.498706 12998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:26.499037 12998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:26.499233 12998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:26.508000 12998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:26.508162 12998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:26.508339 12998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:23:26.520056 12998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:39251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:35061
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:26.521448 12998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:26.522928 12998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:26.537824 13004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:26.538223 13005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:26.538784 13007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:26.539261 12998 server_base.cc:1034] running on GCE node
I20260809 20:23:26.539820 12998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:26.540761 12998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:26.541961 12998 hybrid_clock.cc:648] HybridClock initialized: now 1786307006541921 us; error 36 us; skew 500 ppm
I20260809 20:23:26.542266 12998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:26 dist-test-slave-rnrw krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307006, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:26.546165 12998 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:23:26.548185 12998 webserver.cc:460] Webserver started at http://127.12.78.3:44135/ using document root <none> and password file <none>
I20260809 20:23:26.549103 12998 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:26.549320 12998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:26.549705 12998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:26.553025 12998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "26b94c9c2d764fe899fa0ee873b87a66"
format_stamp: "Formatted at 2026-08-09 20:23:26 on dist-test-slave-rnrw"
server_key: "c3f5282b92c18ff441cfe1c803878d3d"
server_key_iv: "5b5069c3352bffcd10696459a0e425a7"
server_key_version: "encryptionkey@0"
I20260809 20:23:26.553920 12998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "26b94c9c2d764fe899fa0ee873b87a66"
format_stamp: "Formatted at 2026-08-09 20:23:26 on dist-test-slave-rnrw"
server_key: "c3f5282b92c18ff441cfe1c803878d3d"
server_key_iv: "5b5069c3352bffcd10696459a0e425a7"
server_key_version: "encryptionkey@0"
I20260809 20:23:26.560091 12998 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 20:23:26.564632 13015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:26.566000 12998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:23:26.566136 12998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "26b94c9c2d764fe899fa0ee873b87a66"
format_stamp: "Formatted at 2026-08-09 20:23:26 on dist-test-slave-rnrw"
server_key: "c3f5282b92c18ff441cfe1c803878d3d"
server_key_iv: "5b5069c3352bffcd10696459a0e425a7"
server_key_version: "encryptionkey@0"
I20260809 20:23:26.566327 12998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:26.586688 12998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:26.591441 12998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:26.591840 12998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:26.593302 12998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:26.594911 12998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:26.595022 12998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:26.595194 12998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:26.595247 12998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:26.658073 12998 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:33755
I20260809 20:23:26.658277 13137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:33755 every 8 connection(s)
I20260809 20:23:26.659696 12998 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:23:26.665282 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 12998
I20260809 20:23:26.665512 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:23:26.666245 12600 external_mini_cluster.cc:1428] Setting key e9df0201b8eba5de6be5cbe229ada717
Aug 09 20:23:26 dist-test-slave-rnrw krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307006, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:26.684350 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:26.662664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:33775 (local address 127.12.78.62:35061)
0809 20:23:26.663352 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:26.663356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:26.664769 (+ 1413us) server_negotiation.cc:407] Connection header received
0809 20:23:26.668114 (+ 3345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:26.668120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:26.668265 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:26.668386 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:26.670157 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:26.671322 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:26.673205 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:26.673619 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:26.673766 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:26.677890 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:26.677895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:26.677898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:26.677930 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:26.680765 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:26.681625 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:26.681633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:26.681638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:26.681858 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:26.682533 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:26.682536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:26.682539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:26.682913 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:26.683065 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:26.683888 (+ 823us) server_negotiation.cc:299] Negotiation successful
0809 20:23:26.684063 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":427,"threads_started":1}
I20260809 20:23:26.685647 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:26.663513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:35061 (local address 127.12.78.3:33775)
0809 20:23:26.664596 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:26.664654 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:26.667823 (+ 3169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:26.668687 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:26.668695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:26.669328 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:26.669984 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:26.670002 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:26.671491 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:26.671495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:26.673046 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:26.673059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:26.673910 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:26.673921 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:26.674150 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:26.674880 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:26.674908 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:26.677634 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:26.680899 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:26.680908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:26.680924 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:26.681475 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:26.682019 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:26.682023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:26.682025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:26.682410 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:26.683044 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:26.683052 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:26.683474 (+ 422us) client_negotiation.cc:769] Sending connection context
0809 20:23:26.683820 (+ 346us) client_negotiation.cc:240] Negotiation successful
0809 20:23:26.684280 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":624,"threads_started":1}
I20260809 20:23:26.687186 13138 heartbeater.cc:347] Connected to a master server at 127.12.78.62:35061
I20260809 20:23:26.687597 13138 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:26.688603 13138 heartbeater.cc:511] Master 127.12.78.62:35061 requested a full tablet report, sending...
I20260809 20:23:26.690188 12656 ts_manager.cc:194] Registered new tserver with Master: 26b94c9c2d764fe899fa0ee873b87a66 (127.12.78.3:33755)
I20260809 20:23:26.691140 12656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:33775
I20260809 20:23:26.695763 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 20:23:26 dist-test-slave-rnrw krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 20:23:26 dist-test-slave-rnrw krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:23:26.751087 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:26.729682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.12.78.1:35183)
0809 20:23:26.731673 (+ 1991us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:26.731680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:26.731706 (+ 26us) server_negotiation.cc:407] Connection header received
0809 20:23:26.731769 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:26.731776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:26.732023 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:26.732154 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:26.734287 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:26.735455 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:26.737037 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:26.737792 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:26.737893 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:26.743503 (+ 5610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:26.743518 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:26.743534 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:26.743594 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:26.747543 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:26.748393 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:26.748401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:26.748403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:26.748641 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:26.749138 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:26.749143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:26.749148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:26.749562 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:26.750025 (+ 463us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:26.750336 (+ 311us) server_negotiation.cc:299] Negotiation successful
0809 20:23:26.750668 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1782,"thread_start_us":1548,"threads_started":1}
W20260809 20:23:26.752439 12763 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45688
Aug 09 20:23:26 dist-test-slave-rnrw krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 20:23:26 dist-test-slave-rnrw krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:23:26.806967 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:26.792196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.12.78.1:35183)
0809 20:23:26.792495 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:26.792502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:26.792800 (+ 298us) server_negotiation.cc:407] Connection header received
0809 20:23:26.792943 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:26.792947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:26.793015 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:26.793102 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:26.794718 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:26.795397 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:26.796694 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:26.797093 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:26.797193 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:26.801789 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:26.801794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:26.801797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:26.801831 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:26.804347 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:26.805060 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:26.805066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:26.805068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:26.805204 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:26.805590 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:26.805594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:26.805596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:26.805841 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:26.806088 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:26.806410 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 20:23:26.806640 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260809 20:23:26.808522 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 12718
I20260809 20:23:26.822856 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:26.809500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45712 (local address 127.12.78.1:35183)
0809 20:23:26.809713 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:26.809719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:26.809946 (+ 227us) server_negotiation.cc:407] Connection header received
0809 20:23:26.810042 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:26.810046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:26.810098 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:26.810171 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:26.811522 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:26.812186 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:26.813727 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:26.814141 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:26.814238 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:26.816751 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:26.816763 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:26.816766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:26.816797 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:26.819655 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:26.820404 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:26.820412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:26.820415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:26.820686 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:26.821164 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:26.821172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:26.821176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:26.821517 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:26.821862 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:26.822204 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 20:23:26.822431 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 20:23:26.823675 12847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:26.909430 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 12718
I20260809 20:23:26.928874 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 12859
Aug 09 20:23:26 dist-test-slave-rnrw krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:23:26.949890 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:26.929832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.12.78.2:32943)
0809 20:23:26.930508 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:26.930515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:26.930542 (+ 27us) server_negotiation.cc:407] Connection header received
0809 20:23:26.930612 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:26.930619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:26.930828 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:26.930994 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:26.932085 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:26.933369 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:26.934580 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:26.935107 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:26.935219 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:26.939237 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:26.939248 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:26.939261 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:26.939293 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:26.947325 (+ 8032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:26.947989 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:26.947996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:26.947998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:26.948122 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:26.948466 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:26.948469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:26.948471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:26.948891 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:26.949120 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:26.949402 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 20:23:26.949598 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":369,"threads_started":1}
I20260809 20:23:26.951160 12986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:27.033078 12992 heartbeater.cc:503] Master 127.12.78.62:35061 was elected leader, sending a full tablet report...
I20260809 20:23:27.034405 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 12859
W20260809 20:23:27.052812 12642 connection.cc:535] server connection from 127.12.78.2:48021 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 20:23:27.053143 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 12998
Aug 09 20:23:27 dist-test-slave-rnrw krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:23:27.069972 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:27.054070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.12.78.3:33755)
0809 20:23:27.054694 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:27.054702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:27.054732 (+ 30us) server_negotiation.cc:407] Connection header received
0809 20:23:27.054792 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:27.054799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:27.055027 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:27.055164 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:27.056366 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:27.057563 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:27.058823 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:27.059638 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:27.059741 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:27.064367 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:27.064381 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:27.064396 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:27.064437 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:27.067225 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:27.067950 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:27.067957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:27.067959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:27.068083 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:27.068428 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:27.068431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:27.068433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:27.068853 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:27.069094 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:27.069370 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 20:23:27.069574 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":303,"threads_started":1}
I20260809 20:23:27.071259 13133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:27.155264 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 12998
I20260809 20:23:27.173107 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 12622
I20260809 20:23:27.174389 12687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:27.279139 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 12622
2026-08-09T20:23:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5441 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:23:27 dist-test-slave-rnrw krb5kdc[13161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:23:27 dist-test-slave-rnrw krb5kdc[13161](info): set up 2 sockets
Aug 09 20:23:27 dist-test-slave-rnrw krb5kdc[13161](info): commencing operation
krb5kdc: starting...
W20260809 20:23:28.396380 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:23:28 dist-test-slave-rnrw krb5kdc[13161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:23:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:23:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:23:28.595638 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:46687
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:37771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:46687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:28.917637 13175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:28.918004 13175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:28.918193 13175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:28.925688 13175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:23:28.925836 13175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:28.925948 13175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:28.926044 13175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:23:28.926129 13175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:23:28.936022 13175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:46687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:46687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:28.937414 13175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:28.938697 13175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:28.950347 13181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:28.950300 13182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:28.950424 13184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:28.951047 13175 server_base.cc:1034] running on GCE node
I20260809 20:23:28.951804 13175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:28.953212 13175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:28.954474 13175 hybrid_clock.cc:648] HybridClock initialized: now 1786307008954416 us; error 53 us; skew 500 ppm
I20260809 20:23:28.954864 13175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:28 dist-test-slave-rnrw krb5kdc[13161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307008, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:28.958145 13175 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:23:28.959641 13175 webserver.cc:460] Webserver started at http://127.12.78.62:37569/ using document root <none> and password file <none>
I20260809 20:23:28.960384 13175 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:28.960585 13175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:28.960913 13175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:28.963997 13175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "225d079d01ac4638a06bcd1737855b56"
format_stamp: "Formatted at 2026-08-09 20:23:28 on dist-test-slave-rnrw"
server_key: "6a034045a3efade24f812aa4ea123ae8"
server_key_iv: "a390b8a5727fe39a5dd4273a42c3a43d"
server_key_version: "encryptionkey@0"
I20260809 20:23:28.964890 13175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "225d079d01ac4638a06bcd1737855b56"
format_stamp: "Formatted at 2026-08-09 20:23:28 on dist-test-slave-rnrw"
server_key: "6a034045a3efade24f812aa4ea123ae8"
server_key_iv: "a390b8a5727fe39a5dd4273a42c3a43d"
server_key_version: "encryptionkey@0"
I20260809 20:23:28.970265 13175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:23:28.973809 13191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:28.975178 13175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:23:28.975389 13175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "225d079d01ac4638a06bcd1737855b56"
format_stamp: "Formatted at 2026-08-09 20:23:28 on dist-test-slave-rnrw"
server_key: "6a034045a3efade24f812aa4ea123ae8"
server_key_iv: "a390b8a5727fe39a5dd4273a42c3a43d"
server_key_version: "encryptionkey@0"
I20260809 20:23:28.975625 13175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:28.993398 13175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:28.997905 13175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:28.998242 13175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:29.018271 13175 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:46687
I20260809 20:23:29.018306 13246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:46687 every 8 connection(s)
I20260809 20:23:29.019825 13175 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:23:29.023162 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 13175
I20260809 20:23:29.023432 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:23:29.023900 13247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:29.024235 12600 external_mini_cluster.cc:1428] Setting key 40296a6f89c587c865ab008ec03810c2
I20260809 20:23:29.038550 13247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56: Bootstrap starting.
Aug 09 20:23:29 dist-test-slave-rnrw krb5kdc[13161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:29.041843 13247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:29.042904 13247 log.cc:824] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:29.045861 13247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56: No bootstrap required, opened a new log
I20260809 20:23:29.048169 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:29.026309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.12.78.62:46687)
0809 20:23:29.027200 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:29.027212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:29.027258 (+ 46us) server_negotiation.cc:407] Connection header received
0809 20:23:29.029007 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:29.029033 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:29.029436 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:29.029788 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:29.031649 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:29.033145 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:29.034981 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:29.035681 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:29.035819 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:29.040720 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:29.040728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:29.040742 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:29.040783 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:29.044059 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:29.044838 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:29.044850 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:29.044855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:29.045020 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:29.045407 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:29.045412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:29.045415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:29.045932 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:29.046197 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:29.046766 (+ 569us) server_negotiation.cc:299] Negotiation successful
0809 20:23:29.047129 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":422,"threads_started":1}
I20260809 20:23:29.053929 13247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225d079d01ac4638a06bcd1737855b56" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46687 } }
I20260809 20:23:29.054330 13247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:29.054441 13247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225d079d01ac4638a06bcd1737855b56, State: Initialized, Role: FOLLOWER
I20260809 20:23:29.054987 13247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225d079d01ac4638a06bcd1737855b56" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46687 } }
I20260809 20:23:29.055243 13247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:23:29.055392 13247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:23:29.055564 13247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:29.058137 13247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225d079d01ac4638a06bcd1737855b56" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46687 } }
I20260809 20:23:29.058653 13247 leader_election.cc:302] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 225d079d01ac4638a06bcd1737855b56; no voters:
I20260809 20:23:29.059260 13247 leader_election.cc:288] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:23:29.059394 13252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:29.060156 13252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [term 1 LEADER]: Becoming Leader. State: Replica: 225d079d01ac4638a06bcd1737855b56, State: Running, Role: LEADER
I20260809 20:23:29.060763 13252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225d079d01ac4638a06bcd1737855b56" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46687 } }
I20260809 20:23:29.061596 13247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:23:29.064258 13253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "225d079d01ac4638a06bcd1737855b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225d079d01ac4638a06bcd1737855b56" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46687 } } }
I20260809 20:23:29.064378 13254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 225d079d01ac4638a06bcd1737855b56. Latest consensus state: current_term: 1 leader_uuid: "225d079d01ac4638a06bcd1737855b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225d079d01ac4638a06bcd1737855b56" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46687 } } }
I20260809 20:23:29.064862 13254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:29.065407 13253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56 [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:29.069231 13259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:23:29.074035 13259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:23:29.082772 13259 catalog_manager.cc:1347] Generated new cluster ID: 9f181178732b4ff9ac93418ebda93372
I20260809 20:23:29.082899 13259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:23:29.128229 13259 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:23:29.129638 13259 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:23:29.135161 13259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 225d079d01ac4638a06bcd1737855b56: Generated new TSK 0
I20260809 20:23:29.135967 13259 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:23:29.190163 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:46687
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:37771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:29.534145 13274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:29.534432 13274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:29.534605 13274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:29.541857 13274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:29.541965 13274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:29.542174 13274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:23:29.552295 13274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:46687
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:29.553610 13274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:29.554932 13274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:29.568475 13279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:29.568452 13280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:29.568801 13282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:29.569522 13274 server_base.cc:1034] running on GCE node
I20260809 20:23:29.570174 13274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:29.571053 13274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:29.572256 13274 hybrid_clock.cc:648] HybridClock initialized: now 1786307009572215 us; error 31 us; skew 500 ppm
I20260809 20:23:29.572530 13274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:29 dist-test-slave-rnrw krb5kdc[13161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307009, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:29.575667 13274 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:23:29.577138 13274 webserver.cc:460] Webserver started at http://127.12.78.1:40281/ using document root <none> and password file <none>
I20260809 20:23:29.577807 13274 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:29.577907 13274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:29.578171 13274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:29.581333 13274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "faf3c8c961ea4800a47e8420cd82445f"
format_stamp: "Formatted at 2026-08-09 20:23:29 on dist-test-slave-rnrw"
server_key: "fe52afd11f96c4907e0493327ce01c20"
server_key_iv: "062f9db729ad75f51d7c34e3df6f84fb"
server_key_version: "encryptionkey@0"
I20260809 20:23:29.582082 13274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "faf3c8c961ea4800a47e8420cd82445f"
format_stamp: "Formatted at 2026-08-09 20:23:29 on dist-test-slave-rnrw"
server_key: "fe52afd11f96c4907e0493327ce01c20"
server_key_iv: "062f9db729ad75f51d7c34e3df6f84fb"
server_key_version: "encryptionkey@0"
I20260809 20:23:29.587095 13274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:23:29.590636 13289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:29.592031 13274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 20:23:29.592191 13274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "faf3c8c961ea4800a47e8420cd82445f"
format_stamp: "Formatted at 2026-08-09 20:23:29 on dist-test-slave-rnrw"
server_key: "fe52afd11f96c4907e0493327ce01c20"
server_key_iv: "062f9db729ad75f51d7c34e3df6f84fb"
server_key_version: "encryptionkey@0"
I20260809 20:23:29.592376 13274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:29.620236 13274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:29.624485 13274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:29.624822 13274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:29.625808 13274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:29.627125 13274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:29.627208 13274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:29.627307 13274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:29.627352 13274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:29.661227 13274 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:44249
I20260809 20:23:29.661305 13407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:44249 every 8 connection(s)
I20260809 20:23:29.662688 13274 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:23:29.672374 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 13274
I20260809 20:23:29.672638 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:23:29.673437 12600 external_mini_cluster.cc:1428] Setting key d47885fb35bceeba542eb91856ca360a
Aug 09 20:23:29 dist-test-slave-rnrw krb5kdc[13161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307009, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:29.683965 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:29.665329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:58827 (local address 127.12.78.62:46687)
0809 20:23:29.665770 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:29.665774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:29.666711 (+ 937us) server_negotiation.cc:407] Connection header received
0809 20:23:29.668661 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:29.668665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:29.668733 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:29.668879 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:29.670321 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:29.671042 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:29.672507 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:29.673164 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:29.673325 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:29.677097 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:29.677106 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:29.677114 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:29.677160 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:29.680969 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:29.681681 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:29.681689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:29.681695 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:29.681848 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:29.682278 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:29.682286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:29.682291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:29.682661 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:29.682812 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:29.683487 (+ 675us) server_negotiation.cc:299] Negotiation successful
0809 20:23:29.683709 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":219,"threads_started":1}
I20260809 20:23:29.685397 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:29.665868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:46687 (local address 127.12.78.1:58827)
0809 20:23:29.666545 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:29.666583 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:29.668351 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:29.669109 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:29.669117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:29.669619 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:29.670157 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:29.670185 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:29.671161 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:29.671164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:29.672350 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:29.672367 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:29.673381 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:29.673385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:29.673632 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:29.674320 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:29.674355 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:29.676870 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:29.681114 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:29.681121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:29.681134 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:29.681553 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:29.681964 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:29.681967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:29.681969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:29.682142 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:29.682790 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:29.682796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:29.683093 (+ 297us) client_negotiation.cc:769] Sending connection context
0809 20:23:29.683340 (+ 247us) client_negotiation.cc:240] Negotiation successful
0809 20:23:29.683923 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":271,"threads_started":1}
I20260809 20:23:29.686959 13408 heartbeater.cc:347] Connected to a master server at 127.12.78.62:46687
I20260809 20:23:29.687286 13408 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:29.687995 13408 heartbeater.cc:511] Master 127.12.78.62:46687 requested a full tablet report, sending...
I20260809 20:23:29.690129 13209 ts_manager.cc:194] Registered new tserver with Master: faf3c8c961ea4800a47e8420cd82445f (127.12.78.1:44249)
I20260809 20:23:29.691553 13209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:58827
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:23:29.720618 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:46687
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:37771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:30.057335 13414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:30.057667 13414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:30.057837 13414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:30.065354 13414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:30.065510 13414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:30.065683 13414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:23:30.076687 13414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:46687
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:30.078081 13414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:30.079564 13414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:30.093725 13419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:30.093796 13422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:30.093755 13420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:30.094863 13414 server_base.cc:1034] running on GCE node
I20260809 20:23:30.095569 13414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:30.096942 13414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:30.098237 13414 hybrid_clock.cc:648] HybridClock initialized: now 1786307010098166 us; error 70 us; skew 500 ppm
I20260809 20:23:30.098614 13414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:30 dist-test-slave-rnrw krb5kdc[13161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307010, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:30.102002 13414 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:23:30.103485 13414 webserver.cc:460] Webserver started at http://127.12.78.2:38335/ using document root <none> and password file <none>
I20260809 20:23:30.104189 13414 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:30.104323 13414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:30.104718 13414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:30.107641 13414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "49ec8e44ac5647899a2161974018e705"
format_stamp: "Formatted at 2026-08-09 20:23:30 on dist-test-slave-rnrw"
server_key: "efdd07da70ecea421162261426a31fcc"
server_key_iv: "f0e4e3a7f67dc4e2e806b0485f5cf789"
server_key_version: "encryptionkey@0"
I20260809 20:23:30.108429 13414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "49ec8e44ac5647899a2161974018e705"
format_stamp: "Formatted at 2026-08-09 20:23:30 on dist-test-slave-rnrw"
server_key: "efdd07da70ecea421162261426a31fcc"
server_key_iv: "f0e4e3a7f67dc4e2e806b0485f5cf789"
server_key_version: "encryptionkey@0"
I20260809 20:23:30.113770 13414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 20:23:30.117297 13429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:30.118566 13414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:23:30.118762 13414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "49ec8e44ac5647899a2161974018e705"
format_stamp: "Formatted at 2026-08-09 20:23:30 on dist-test-slave-rnrw"
server_key: "efdd07da70ecea421162261426a31fcc"
server_key_iv: "f0e4e3a7f67dc4e2e806b0485f5cf789"
server_key_version: "encryptionkey@0"
I20260809 20:23:30.118969 13414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:30.139037 13414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:30.143550 13414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:30.143869 13414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:30.144917 13414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:30.146301 13414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:30.146405 13414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:30.146553 13414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:30.146627 13414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:30.182029 13414 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:43779
I20260809 20:23:30.182119 13548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:43779 every 8 connection(s)
I20260809 20:23:30.183365 13414 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:23:30.190078 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 13414
I20260809 20:23:30.190300 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:23:30.190737 12600 external_mini_cluster.cc:1428] Setting key c5f72df05ac6c0683b480c3e0c8935e6
Aug 09 20:23:30 dist-test-slave-rnrw krb5kdc[13161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307010, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:30.205255 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:30.186095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:38283 (local address 127.12.78.62:46687)
0809 20:23:30.186635 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:30.186640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:30.188179 (+ 1539us) server_negotiation.cc:407] Connection header received
0809 20:23:30.189759 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:30.189765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:30.189876 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:30.190008 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:30.191862 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.192807 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.194402 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.194855 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.195025 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:30.199081 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:30.199086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:30.199090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:30.199124 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:30.202061 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:30.202815 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:30.202822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:30.202830 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:30.202972 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:30.203425 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:30.203431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:30.203434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:30.203726 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:30.203890 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:30.204617 (+ 727us) server_negotiation.cc:299] Negotiation successful
0809 20:23:30.204851 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":275,"threads_started":1}
I20260809 20:23:30.206288 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:30.186498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:46687 (local address 127.12.78.2:38283)
0809 20:23:30.187301 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:30.187367 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:30.189469 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:30.190299 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:30.190308 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:30.190955 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:30.191683 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:30.191705 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.192993 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.192997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:30.194251 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:30.194264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.195106 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.195113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:30.195356 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:30.196111 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:30.196152 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:30.198833 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:30.202216 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:30.202226 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:30.202243 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:30.202680 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:30.203101 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:30.203106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:30.203110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:30.203309 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:30.203864 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:30.203872 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:30.204229 (+ 357us) client_negotiation.cc:769] Sending connection context
0809 20:23:30.204508 (+ 279us) client_negotiation.cc:240] Negotiation successful
0809 20:23:30.205111 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":290,"threads_started":1}
I20260809 20:23:30.208000 13550 heartbeater.cc:347] Connected to a master server at 127.12.78.62:46687
I20260809 20:23:30.208321 13550 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:30.209159 13550 heartbeater.cc:511] Master 127.12.78.62:46687 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
I20260809 20:23:30.210914 13209 ts_manager.cc:194] Registered new tserver with Master: 49ec8e44ac5647899a2161974018e705 (127.12.78.2:43779)
I20260809 20:23:30.211838 13209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:38283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:23:30.237131 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:46687
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:37771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:30.572151 13556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:30.572422 13556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:30.572518 13556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:30.579634 13556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:30.579746 13556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:30.579871 13556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:23:30.589756 13556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:46687
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:30.591017 13556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:30.592298 13556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:30.605556 13564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:30.605654 13561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:30.605651 13562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:30.606566 13556 server_base.cc:1034] running on GCE node
I20260809 20:23:30.607129 13556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:30.608006 13556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:30.609262 13556 hybrid_clock.cc:648] HybridClock initialized: now 1786307010609198 us; error 56 us; skew 500 ppm
I20260809 20:23:30.609548 13556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:30 dist-test-slave-rnrw krb5kdc[13161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307010, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:30.613050 13556 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:23:30.614622 13556 webserver.cc:460] Webserver started at http://127.12.78.3:33621/ using document root <none> and password file <none>
I20260809 20:23:30.615353 13556 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:30.615491 13556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:30.615832 13556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:30.618911 13556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "acf3bf6f3c0543a294f3f18dc29d00c7"
format_stamp: "Formatted at 2026-08-09 20:23:30 on dist-test-slave-rnrw"
server_key: "4635dae87751e78ce24cc8154a3da30d"
server_key_iv: "9c08c669b36656f4df48395c8b3c288a"
server_key_version: "encryptionkey@0"
I20260809 20:23:30.619757 13556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "acf3bf6f3c0543a294f3f18dc29d00c7"
format_stamp: "Formatted at 2026-08-09 20:23:30 on dist-test-slave-rnrw"
server_key: "4635dae87751e78ce24cc8154a3da30d"
server_key_iv: "9c08c669b36656f4df48395c8b3c288a"
server_key_version: "encryptionkey@0"
I20260809 20:23:30.625404 13556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 20:23:30.628945 13571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:30.630285 13556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 20:23:30.630457 13556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "acf3bf6f3c0543a294f3f18dc29d00c7"
format_stamp: "Formatted at 2026-08-09 20:23:30 on dist-test-slave-rnrw"
server_key: "4635dae87751e78ce24cc8154a3da30d"
server_key_iv: "9c08c669b36656f4df48395c8b3c288a"
server_key_version: "encryptionkey@0"
I20260809 20:23:30.630667 13556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:30.651401 13556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:30.655627 13556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:30.655938 13556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:30.657055 13556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:30.658433 13556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:30.658509 13556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:30.658672 13556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:30.658727 13556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:30.693519 13556 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:43855
I20260809 20:23:30.693567 13690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:43855 every 8 connection(s)
I20260809 20:23:30.694272 13408 heartbeater.cc:503] Master 127.12.78.62:46687 was elected leader, sending a full tablet report...
I20260809 20:23:30.694914 13556 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:23:30.696893 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 13556
I20260809 20:23:30.697088 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:23:30.697646 12600 external_mini_cluster.cc:1428] Setting key 6c1ff0c25d7bcda6c866e23f60178927
Aug 09 20:23:30 dist-test-slave-rnrw krb5kdc[13161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307010, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:30.715137 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:30.697794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:46177 (local address 127.12.78.62:46687)
0809 20:23:30.697977 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:30.697982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:30.699628 (+ 1646us) server_negotiation.cc:407] Connection header received
0809 20:23:30.701547 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:30.701551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:30.701617 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:30.701709 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:30.703030 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.703745 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.705277 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.705687 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.705780 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:30.709263 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:30.709267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:30.709270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:30.709302 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:30.712118 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:30.713061 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:30.713065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:30.713067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:30.713251 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:30.713642 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:30.713645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:30.713647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:30.713921 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:30.714038 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:30.714740 (+ 702us) server_negotiation.cc:299] Negotiation successful
0809 20:23:30.714907 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 20:23:30.716105 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:30.698413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:46687 (local address 127.12.78.3:46177)
0809 20:23:30.699463 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:30.699507 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:30.701302 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:30.701870 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:30.701877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:30.702365 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:30.702892 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:30.702908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.703863 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.703866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:30.705156 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:30.705168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.705867 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.705871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:30.706055 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:30.706756 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:30.706797 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:30.709066 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:30.712309 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:30.712318 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:30.712335 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:30.712928 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:30.713353 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:30.713357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:30.713359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:30.713543 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:30.714031 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:30.714037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:30.714340 (+ 303us) client_negotiation.cc:769] Sending connection context
0809 20:23:30.714597 (+ 257us) client_negotiation.cc:240] Negotiation successful
0809 20:23:30.714995 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":558,"threads_started":1}
I20260809 20:23:30.717744 13691 heartbeater.cc:347] Connected to a master server at 127.12.78.62:46687
I20260809 20:23:30.718040 13691 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:30.718729 13691 heartbeater.cc:511] Master 127.12.78.62:46687 requested a full tablet report, sending...
I20260809 20:23:30.720194 13209 ts_manager.cc:194] Registered new tserver with Master: acf3bf6f3c0543a294f3f18dc29d00c7 (127.12.78.3:43855)
I20260809 20:23:30.721066 13209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:46177
I20260809 20:23:30.725584 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:23:30.747745 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:30.732373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53274 (local address 127.12.78.62:46687)
0809 20:23:30.732586 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:30.732595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:30.733357 (+ 762us) server_negotiation.cc:407] Connection header received
0809 20:23:30.733600 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:30.733606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:30.733721 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:30.733858 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:30.735886 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.736742 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.738221 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.738616 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.738750 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:30.741596 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:30.741601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:30.741604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:30.741637 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:30.744312 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:30.745096 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:30.745100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:30.745102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:30.745285 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:30.745784 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:30.745790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:30.745796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:30.746205 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:30.746564 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:30.746933 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 20:23:30.747230 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 20:23:30.754650 13209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:23:30.763546 13209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:23:30.794507 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:30.778303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.12.78.1:44249)
0809 20:23:30.784864 (+ 6561us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:30.784872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:30.784901 (+ 29us) server_negotiation.cc:407] Connection header received
0809 20:23:30.784997 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:30.785004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:30.785310 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:30.785509 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:30.786351 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.788326 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.790743 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.792921 (+ 2178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.793060 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:30.793434 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:30.793766 (+ 332us) server_negotiation.cc:299] Negotiation successful
0809 20:23:30.793955 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6240,"thread_start_us":5659,"threads_started":1}
I20260809 20:23:30.794512 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:30.777208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:44249 (local address 127.0.0.1:57942)
0809 20:23:30.783445 (+ 6237us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:30.783486 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:30.783746 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:30.785540 (+ 1794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:30.785547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:30.785623 (+ 76us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:30.786195 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:30.786204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.788480 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.788485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:30.790589 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:30.790600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.793186 (+ 2586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.793194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:30.793535 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:30.793572 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 20:23:30.793724 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 20:23:30.793918 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5941,"thread_start_us":7933,"threads_started":1}
I20260809 20:23:30.796826 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:30.777421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:43779 (local address 127.0.0.1:40262)
0809 20:23:30.785431 (+ 8010us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:30.785453 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:30.785654 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:30.786500 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:30.786506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:30.786533 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:30.787216 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:30.787241 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.790097 (+ 2856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.790110 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:30.792982 (+ 2872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:30.792996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.795571 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.795579 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:30.795748 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:30.795770 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 20:23:30.795877 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 20:23:30.796004 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7898,"thread_start_us":5508,"threads_started":1}
I20260809 20:23:30.797312 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:30.778100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.12.78.2:43779)
0809 20:23:30.784848 (+ 6748us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:30.784855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:30.785544 (+ 689us) server_negotiation.cc:407] Connection header received
0809 20:23:30.785937 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:30.785945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:30.786255 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:30.786498 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:30.787447 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.789915 (+ 2468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.793216 (+ 3301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.795073 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.795193 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:30.795482 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:30.795919 (+ 437us) server_negotiation.cc:299] Negotiation successful
0809 20:23:30.796778 (+ 859us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6442,"thread_start_us":6005,"threads_started":1}
I20260809 20:23:30.799706 13342 tablet_service.cc:1467] Processing CreateTablet for tablet e362378f770f493caf089c30e74b6766 (DEFAULT_TABLE table=test-table [id=da15f01b0866475c93ef77a53148701a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:30.801517 13342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e362378f770f493caf089c30e74b6766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:30.801818 13708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:30.789416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:43855 (local address 127.0.0.1:56202)
0809 20:23:30.790018 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:30.790035 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:30.790154 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:30.792334 (+ 2180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:30.792339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:30.792370 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:30.792997 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:30.793006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.795344 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.795350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:30.798513 (+ 3163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:30.798529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.800806 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.800813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:30.801082 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:30.801114 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 20:23:30.801184 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 20:23:30.801307 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":497,"threads_started":1}
I20260809 20:23:30.801337 13483 tablet_service.cc:1467] Processing CreateTablet for tablet e362378f770f493caf089c30e74b6766 (DEFAULT_TABLE table=test-table [id=da15f01b0866475c93ef77a53148701a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:30.802721 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:30.786045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56202 (local address 127.12.78.3:43855)
0809 20:23:30.791473 (+ 5428us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:30.791482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:30.791549 (+ 67us) server_negotiation.cc:407] Connection header received
0809 20:23:30.791637 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:30.791644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:30.791908 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:30.792114 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:30.793203 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.795177 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.798720 (+ 3543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:30.800212 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:30.800336 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:30.802125 (+ 1789us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:30.802202 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 20:23:30.802353 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5199,"thread_start_us":4888,"threads_started":1}
I20260809 20:23:30.802847 13483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e362378f770f493caf089c30e74b6766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:30.806733 13621 tablet_service.cc:1467] Processing CreateTablet for tablet e362378f770f493caf089c30e74b6766 (DEFAULT_TABLE table=test-table [id=da15f01b0866475c93ef77a53148701a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:30.808591 13621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e362378f770f493caf089c30e74b6766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:30.815656 13709 tablet_bootstrap.cc:492] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f: Bootstrap starting.
I20260809 20:23:30.819579 13710 tablet_bootstrap.cc:492] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705: Bootstrap starting.
I20260809 20:23:30.820017 13709 tablet_bootstrap.cc:654] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:30.822321 13709 log.cc:824] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:30.824942 13710 tablet_bootstrap.cc:654] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:30.825250 13712 tablet_bootstrap.cc:492] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7: Bootstrap starting.
I20260809 20:23:30.826401 13709 tablet_bootstrap.cc:492] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f: No bootstrap required, opened a new log
I20260809 20:23:30.826613 13710 log.cc:824] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:30.826723 13709 ts_tablet_manager.cc:1397] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260809 20:23:30.829773 13710 tablet_bootstrap.cc:492] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705: No bootstrap required, opened a new log
I20260809 20:23:30.830066 13710 ts_tablet_manager.cc:1397] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260809 20:23:30.830732 13712 tablet_bootstrap.cc:654] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:30.832289 13712 log.cc:824] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:30.836179 13712 tablet_bootstrap.cc:492] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7: No bootstrap required, opened a new log
I20260809 20:23:30.836439 13712 ts_tablet_manager.cc:1397] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260809 20:23:30.839291 13709 raft_consensus.cc:348] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf3c8c961ea4800a47e8420cd82445f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 44249 } } peers { permanent_uuid: "49ec8e44ac5647899a2161974018e705" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43779 } } peers { permanent_uuid: "acf3bf6f3c0543a294f3f18dc29d00c7" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43855 } }
I20260809 20:23:30.839783 13709 raft_consensus.cc:374] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:30.839952 13709 raft_consensus.cc:729] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faf3c8c961ea4800a47e8420cd82445f, State: Initialized, Role: FOLLOWER
I20260809 20:23:30.840689 13709 consensus_queue.cc:260] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf3c8c961ea4800a47e8420cd82445f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 44249 } } peers { permanent_uuid: "49ec8e44ac5647899a2161974018e705" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43779 } } peers { permanent_uuid: "acf3bf6f3c0543a294f3f18dc29d00c7" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43855 } }
I20260809 20:23:30.842742 13709 ts_tablet_manager.cc:1428] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260809 20:23:30.842427 13710 raft_consensus.cc:348] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf3c8c961ea4800a47e8420cd82445f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 44249 } } peers { permanent_uuid: "49ec8e44ac5647899a2161974018e705" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43779 } } peers { permanent_uuid: "acf3bf6f3c0543a294f3f18dc29d00c7" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43855 } }
I20260809 20:23:30.842972 13710 raft_consensus.cc:374] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:30.843140 13710 raft_consensus.cc:729] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ec8e44ac5647899a2161974018e705, State: Initialized, Role: FOLLOWER
I20260809 20:23:30.843887 13710 consensus_queue.cc:260] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf3c8c961ea4800a47e8420cd82445f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 44249 } } peers { permanent_uuid: "49ec8e44ac5647899a2161974018e705" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43779 } } peers { permanent_uuid: "acf3bf6f3c0543a294f3f18dc29d00c7" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43855 } }
I20260809 20:23:30.845821 13710 ts_tablet_manager.cc:1428] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705: Time spent starting tablet: real 0.016s user 0.005s sys 0.011s
I20260809 20:23:30.846203 13550 heartbeater.cc:503] Master 127.12.78.62:46687 was elected leader, sending a full tablet report...
I20260809 20:23:30.849578 13712 raft_consensus.cc:348] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf3c8c961ea4800a47e8420cd82445f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 44249 } } peers { permanent_uuid: "49ec8e44ac5647899a2161974018e705" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43779 } } peers { permanent_uuid: "acf3bf6f3c0543a294f3f18dc29d00c7" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43855 } }
I20260809 20:23:30.850018 13712 raft_consensus.cc:374] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:30.850128 13712 raft_consensus.cc:729] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acf3bf6f3c0543a294f3f18dc29d00c7, State: Initialized, Role: FOLLOWER
I20260809 20:23:30.850744 13712 consensus_queue.cc:260] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf3c8c961ea4800a47e8420cd82445f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 44249 } } peers { permanent_uuid: "49ec8e44ac5647899a2161974018e705" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43779 } } peers { permanent_uuid: "acf3bf6f3c0543a294f3f18dc29d00c7" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43855 } }
I20260809 20:23:30.851964 13691 heartbeater.cc:503] Master 127.12.78.62:46687 was elected leader, sending a full tablet report...
I20260809 20:23:30.852411 13712 ts_tablet_manager.cc:1428] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260809 20:23:30.915175 13409 tablet.cc:2369] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:23:30.935694 13551 tablet.cc:2369] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:23:30.946691 13692 tablet.cc:2369] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:23:31.024528 13716 raft_consensus.cc:482] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:23:31.024912 13716 raft_consensus.cc:504] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf3c8c961ea4800a47e8420cd82445f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 44249 } } peers { permanent_uuid: "49ec8e44ac5647899a2161974018e705" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43779 } } peers { permanent_uuid: "acf3bf6f3c0543a294f3f18dc29d00c7" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43855 } }
I20260809 20:23:31.026990 13716 leader_election.cc:288] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faf3c8c961ea4800a47e8420cd82445f (127.12.78.1:44249), acf3bf6f3c0543a294f3f18dc29d00c7 (127.12.78.3:43855)
I20260809 20:23:31.036305 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:31.027699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:41657 (local address 127.12.78.1:44249)
0809 20:23:31.027929 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:31.027935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:31.028765 (+ 830us) server_negotiation.cc:407] Connection header received
0809 20:23:31.028982 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:31.028987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:31.029056 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:31.029165 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:31.030187 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.030968 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.033533 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.035084 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.035235 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:31.035354 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:31.035856 (+ 502us) server_negotiation.cc:299] Negotiation successful
0809 20:23:31.035994 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 20:23:31.036373 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:31.027494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:44249 (local address 127.12.78.2:41657)
0809 20:23:31.028663 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:31.028685 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:31.028835 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:31.029366 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:31.029373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:31.029401 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:31.030024 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:31.030035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.031102 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.031109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:31.033276 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:31.033303 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.035380 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.035398 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:31.035668 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:31.035704 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 20:23:31.035803 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 20:23:31.035931 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"thread_start_us":761,"threads_started":1}
I20260809 20:23:31.038072 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e362378f770f493caf089c30e74b6766" candidate_uuid: "49ec8e44ac5647899a2161974018e705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faf3c8c961ea4800a47e8420cd82445f" is_pre_election: true
I20260809 20:23:31.038650 13363 raft_consensus.cc:2454] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49ec8e44ac5647899a2161974018e705 in term 0.
I20260809 20:23:31.039389 13433 leader_election.cc:302] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ec8e44ac5647899a2161974018e705, faf3c8c961ea4800a47e8420cd82445f; no voters:
I20260809 20:23:31.039876 13716 raft_consensus.cc:2781] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:23:31.040112 13716 raft_consensus.cc:482] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:23:31.040259 13716 raft_consensus.cc:3037] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:31.041594 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:31.029395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:55325 (local address 127.12.78.3:43855)
0809 20:23:31.029574 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:31.029578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:31.033512 (+ 3934us) server_negotiation.cc:407] Connection header received
0809 20:23:31.033775 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:31.033781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:31.033849 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:31.033947 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:31.034937 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.036325 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.038839 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.040337 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.040477 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:31.040608 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:31.041280 (+ 672us) server_negotiation.cc:299] Negotiation successful
0809 20:23:31.041375 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 20:23:31.041909 13719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:31.028778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:43855 (local address 127.12.78.2:55325)
0809 20:23:31.033425 (+ 4647us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:31.033443 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:31.033581 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:31.033960 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:31.033965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:31.033988 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:31.034753 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:31.034767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.036609 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.036615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:31.038661 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:31.038679 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.040784 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.040792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:31.041088 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:31.041126 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 20:23:31.041254 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 20:23:31.041388 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4430,"thread_start_us":4114,"threads_started":1}
I20260809 20:23:31.043206 13642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e362378f770f493caf089c30e74b6766" candidate_uuid: "49ec8e44ac5647899a2161974018e705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acf3bf6f3c0543a294f3f18dc29d00c7" is_pre_election: true
I20260809 20:23:31.043713 13642 raft_consensus.cc:2454] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49ec8e44ac5647899a2161974018e705 in term 0.
I20260809 20:23:31.044173 13716 raft_consensus.cc:504] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf3c8c961ea4800a47e8420cd82445f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 44249 } } peers { permanent_uuid: "49ec8e44ac5647899a2161974018e705" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43779 } } peers { permanent_uuid: "acf3bf6f3c0543a294f3f18dc29d00c7" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43855 } }
I20260809 20:23:31.045011 13716 leader_election.cc:288] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [CANDIDATE]: Term 1 election: Requested vote from peers faf3c8c961ea4800a47e8420cd82445f (127.12.78.1:44249), acf3bf6f3c0543a294f3f18dc29d00c7 (127.12.78.3:43855)
I20260809 20:23:31.045404 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e362378f770f493caf089c30e74b6766" candidate_uuid: "49ec8e44ac5647899a2161974018e705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faf3c8c961ea4800a47e8420cd82445f"
I20260809 20:23:31.045462 13642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e362378f770f493caf089c30e74b6766" candidate_uuid: "49ec8e44ac5647899a2161974018e705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acf3bf6f3c0543a294f3f18dc29d00c7"
I20260809 20:23:31.045625 13363 raft_consensus.cc:3037] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:31.045724 13642 raft_consensus.cc:3037] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:31.048332 13363 raft_consensus.cc:2454] T e362378f770f493caf089c30e74b6766 P faf3c8c961ea4800a47e8420cd82445f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ec8e44ac5647899a2161974018e705 in term 1.
I20260809 20:23:31.048579 13642 raft_consensus.cc:2454] T e362378f770f493caf089c30e74b6766 P acf3bf6f3c0543a294f3f18dc29d00c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ec8e44ac5647899a2161974018e705 in term 1.
I20260809 20:23:31.048792 13433 leader_election.cc:302] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ec8e44ac5647899a2161974018e705, faf3c8c961ea4800a47e8420cd82445f; no voters:
I20260809 20:23:31.049125 13716 raft_consensus.cc:2781] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:31.049734 13716 raft_consensus.cc:686] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [term 1 LEADER]: Becoming Leader. State: Replica: 49ec8e44ac5647899a2161974018e705, State: Running, Role: LEADER
I20260809 20:23:31.050467 13716 consensus_queue.cc:237] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf3c8c961ea4800a47e8420cd82445f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 44249 } } peers { permanent_uuid: "49ec8e44ac5647899a2161974018e705" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43779 } } peers { permanent_uuid: "acf3bf6f3c0543a294f3f18dc29d00c7" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43855 } }
I20260809 20:23:31.055629 13208 catalog_manager.cc:5495] T e362378f770f493caf089c30e74b6766 P 49ec8e44ac5647899a2161974018e705 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49ec8e44ac5647899a2161974018e705 (127.12.78.2). New cstate: current_term: 1 leader_uuid: "49ec8e44ac5647899a2161974018e705" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf3c8c961ea4800a47e8420cd82445f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 44249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49ec8e44ac5647899a2161974018e705" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acf3bf6f3c0543a294f3f18dc29d00c7" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43855 } health_report { overall_health: UNKNOWN } } }
Aug 09 20:23:31 dist-test-slave-rnrw krb5kdc[13161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:23:31.097486 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:31.079883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.12.78.1:44249)
0809 20:23:31.080095 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:31.080100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:31.080790 (+ 690us) server_negotiation.cc:407] Connection header received
0809 20:23:31.080950 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:31.080953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:31.081064 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:31.081149 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:31.083344 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.084205 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.085478 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.086037 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.086131 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:31.090836 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:31.090852 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:31.090868 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:31.090924 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:31.094377 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:31.095063 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:31.095070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:31.095072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:31.095253 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:31.095683 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:31.095686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:31.095687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:31.096201 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:31.096457 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:31.096812 (+ 355us) server_negotiation.cc:299] Negotiation successful
0809 20:23:31.096968 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Aug 09 20:23:31 dist-test-slave-rnrw krb5kdc[13161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 20:23:31 dist-test-slave-rnrw krb5kdc[13161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:23:31.152025 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:31.137887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.12.78.1:44249)
0809 20:23:31.138107 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:31.138112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:31.138338 (+ 226us) server_negotiation.cc:407] Connection header received
0809 20:23:31.138488 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:31.138493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:31.138564 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:31.138659 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:31.140003 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.140954 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.142308 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.142753 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.142860 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:31.146766 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:31.146771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:31.146774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:31.146809 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:31.149748 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:31.150330 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:31.150333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:31.150336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:31.150470 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:31.150837 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:31.150840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:31.150842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:31.151095 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:31.151282 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:31.151495 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 20:23:31.151661 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Aug 09 20:23:31 dist-test-slave-rnrw krb5kdc[13161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 20:23:31 dist-test-slave-rnrw krb5kdc[13161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:23:31.208460 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:31.193243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.12.78.1:44249)
0809 20:23:31.193519 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:31.193530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:31.194011 (+ 481us) server_negotiation.cc:407] Connection header received
0809 20:23:31.194174 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:31.194179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:31.194249 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:31.194336 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:31.196487 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.197273 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.198749 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.199163 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.199257 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:31.203355 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:31.203360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:31.203363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:31.203398 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:31.206095 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:31.206735 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:31.206739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:31.206741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:31.206857 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:31.207167 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:31.207170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:31.207172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:31.207462 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:31.207654 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:31.207916 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 20:23:31.208097 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260809 20:23:31.210383 13319 tablet_service.cc:2908] Rejecting scan request for tablet e362378f770f493caf089c30e74b6766: Uninitialized: clean time has not yet been initialized
I20260809 20:23:31.214102 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 13274
I20260809 20:23:31.226575 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:31.215262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.12.78.1:44249)
0809 20:23:31.215426 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:31.215430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:31.215865 (+ 435us) server_negotiation.cc:407] Connection header received
0809 20:23:31.215986 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:31.215989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:31.216057 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:31.216134 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:31.217304 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.217965 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.219056 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.219461 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.219560 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:31.221763 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:31.221784 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:31.221787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:31.221827 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:31.224296 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:31.224985 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:31.224988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:31.224991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:31.225100 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:31.225409 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:31.225412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:31.225414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:31.225682 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:31.225883 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:31.226088 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 20:23:31.226246 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 20:23:31.227335 13403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:31.321936 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 13274
I20260809 20:23:31.341373 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 13414
Aug 09 20:23:31 dist-test-slave-rnrw krb5kdc[13161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:23:31.356704 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:31.342155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.12.78.2:43779)
0809 20:23:31.342631 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:31.342636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:31.342659 (+ 23us) server_negotiation.cc:407] Connection header received
0809 20:23:31.342718 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:31.342721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:31.342784 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:31.342882 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:31.344052 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.345035 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.346314 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.346769 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.346917 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:31.351055 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:31.351069 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:31.351084 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:31.351118 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:31.353898 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:31.354556 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:31.354564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:31.354566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:31.354713 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:31.355087 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:31.355094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:31.355097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:31.355450 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:31.355742 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:31.356060 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 20:23:31.356305 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":273,"threads_started":1}
I20260809 20:23:31.357376 13544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:31.449277 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 13414
I20260809 20:23:31.449345 13719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:31.448775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:44249 (local address 127.12.78.2:60517)
0809 20:23:31.448955 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:31.449183 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 20:23:31.470679 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 13556
Aug 09 20:23:31 dist-test-slave-rnrw krb5kdc[13161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:23:31.486200 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:31.471515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.12.78.3:43855)
0809 20:23:31.471687 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:31.471691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:31.471769 (+ 78us) server_negotiation.cc:407] Connection header received
0809 20:23:31.471920 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:31.471924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:31.471990 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:31.472086 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:31.473484 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.474189 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.475560 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:31.475932 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:31.476023 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:31.480637 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:31.480652 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:31.480669 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:31.480708 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:31.483593 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:31.484264 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:31.484272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:31.484274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:31.484405 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:31.484910 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:31.484913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:31.484916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:31.485290 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:31.485506 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:31.485740 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 20:23:31.485917 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 20:23:31.486934 13686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:31.575639 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 13556
I20260809 20:23:31.595213 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 13175
I20260809 20:23:31.596381 13242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:31.705760 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 13175
2026-08-09T20:23:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4426 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:23:31 dist-test-slave-rnrw krb5kdc[13741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:23:31 dist-test-slave-rnrw krb5kdc[13741](info): set up 2 sockets
Aug 09 20:23:31 dist-test-slave-rnrw krb5kdc[13741](info): commencing operation
krb5kdc: starting...
W20260809 20:23:32.851925 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:23:32 dist-test-slave-rnrw krb5kdc[13741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:23:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:23:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:23:33.055723 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:38475
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:38475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:33.398260 13754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:33.398622 13754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:33.398816 13754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:33.406138 13754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:23:33.406286 13754 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 20:23:33.406368 13754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:33.406459 13754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:33.406565 13754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:23:33.406653 13754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:23:33.416671 13754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:38475
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:38475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:33.418185 13754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:33.419572 13754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:33.431738 13759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:33.431762 13762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:33.431819 13760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:33.433015 13754 server_base.cc:1034] running on GCE node
I20260809 20:23:33.433776 13754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:33.435127 13754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:33.436360 13754 hybrid_clock.cc:648] HybridClock initialized: now 1786307013436302 us; error 29 us; skew 500 ppm
I20260809 20:23:33.436781 13754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:33 dist-test-slave-rnrw krb5kdc[13741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307013, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:33.439946 13754 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:23:33.441432 13754 webserver.cc:460] Webserver started at http://127.12.78.62:42649/ using document root <none> and password file <none>
I20260809 20:23:33.442097 13754 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:33.442185 13754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:33.442480 13754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:33.445446 13754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "3a518b8ffed0449f9906c87100f93a5c"
format_stamp: "Formatted at 2026-08-09 20:23:33 on dist-test-slave-rnrw"
server_key: "73aaf62bc9383f32bec08227b8cdcead"
server_key_iv: "b3eb0b99944f61148922ee7418f91fd4"
server_key_version: "encryptionkey@0"
I20260809 20:23:33.446159 13754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "3a518b8ffed0449f9906c87100f93a5c"
format_stamp: "Formatted at 2026-08-09 20:23:33 on dist-test-slave-rnrw"
server_key: "73aaf62bc9383f32bec08227b8cdcead"
server_key_iv: "b3eb0b99944f61148922ee7418f91fd4"
server_key_version: "encryptionkey@0"
I20260809 20:23:33.451220 13754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:23:33.454610 13769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:33.455937 13754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 20:23:33.456120 13754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "3a518b8ffed0449f9906c87100f93a5c"
format_stamp: "Formatted at 2026-08-09 20:23:33 on dist-test-slave-rnrw"
server_key: "73aaf62bc9383f32bec08227b8cdcead"
server_key_iv: "b3eb0b99944f61148922ee7418f91fd4"
server_key_version: "encryptionkey@0"
I20260809 20:23:33.456339 13754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:33.490091 13754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:33.494608 13754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:33.494900 13754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:33.518414 13754 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:38475
I20260809 20:23:33.518483 13831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:38475 every 8 connection(s)
I20260809 20:23:33.519933 13754 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:23:33.523794 13832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:33.524427 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 13754
I20260809 20:23:33.524698 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:23:33.525382 12600 external_mini_cluster.cc:1428] Setting key 5980dc01e312151894eaa80d92e7e487
I20260809 20:23:33.537639 13832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c: Bootstrap starting.
Aug 09 20:23:33 dist-test-slave-rnrw krb5kdc[13741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:33.541313 13832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:33.542541 13832 log.cc:824] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:33.545449 13832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c: No bootstrap required, opened a new log
I20260809 20:23:33.548147 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:33.527312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.12.78.62:38475)
0809 20:23:33.528143 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:33.528156 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:33.528196 (+ 40us) server_negotiation.cc:407] Connection header received
0809 20:23:33.529905 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:33.529926 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:33.530298 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:33.530614 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:33.532420 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:33.533726 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:33.535551 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:33.536209 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:33.536338 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:33.541040 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:33.541047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:33.541061 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:33.541096 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:33.544130 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:33.544838 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:33.544846 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:33.544854 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:33.545037 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:33.545414 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:33.545420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:33.545423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:33.546002 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:33.546305 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:33.546740 (+ 435us) server_negotiation.cc:299] Negotiation successful
0809 20:23:33.547163 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":430,"threads_started":1}
I20260809 20:23:33.553447 13832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a518b8ffed0449f9906c87100f93a5c" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 38475 } }
I20260809 20:23:33.553804 13832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:33.553921 13832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a518b8ffed0449f9906c87100f93a5c, State: Initialized, Role: FOLLOWER
I20260809 20:23:33.554579 13832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a518b8ffed0449f9906c87100f93a5c" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 38475 } }
I20260809 20:23:33.554849 13832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:23:33.555016 13832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:23:33.555217 13832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:33.557462 13832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a518b8ffed0449f9906c87100f93a5c" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 38475 } }
I20260809 20:23:33.557905 13832 leader_election.cc:302] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a518b8ffed0449f9906c87100f93a5c; no voters:
I20260809 20:23:33.558403 13832 leader_election.cc:288] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:23:33.558554 13837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:33.559900 13837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [term 1 LEADER]: Becoming Leader. State: Replica: 3a518b8ffed0449f9906c87100f93a5c, State: Running, Role: LEADER
I20260809 20:23:33.560509 13837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a518b8ffed0449f9906c87100f93a5c" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 38475 } }
I20260809 20:23:33.560772 13832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:23:33.563575 13838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a518b8ffed0449f9906c87100f93a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a518b8ffed0449f9906c87100f93a5c" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 38475 } } }
I20260809 20:23:33.563619 13839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a518b8ffed0449f9906c87100f93a5c. Latest consensus state: current_term: 1 leader_uuid: "3a518b8ffed0449f9906c87100f93a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a518b8ffed0449f9906c87100f93a5c" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 38475 } } }
I20260809 20:23:33.563979 13838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:33.564002 13839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:33.566917 13844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:23:33.572423 13844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:23:33.581573 13844 catalog_manager.cc:1347] Generated new cluster ID: 41d4e7d0304243d0b8ed09be589f301e
I20260809 20:23:33.581698 13844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:23:33.599309 13844 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:23:33.601198 13844 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:23:33.609052 13844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c: Generated new TSK 0
I20260809 20:23:33.610105 13844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:23:33.660822 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:38475
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:33.995803 13858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:33.996166 13858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:33.996378 13858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:34.004220 13858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:34.004390 13858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:34.004585 13858 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 20:23:34.004748 13858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:23:34.014997 13858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.78.62:38475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:34.016376 13858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:34.017760 13858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:34.031598 13864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:34.031539 13866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:34.031620 13863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:34.032375 13858 server_base.cc:1034] running on GCE node
I20260809 20:23:34.033020 13858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:34.033936 13858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:34.035204 13858 hybrid_clock.cc:648] HybridClock initialized: now 1786307014035138 us; error 53 us; skew 500 ppm
I20260809 20:23:34.035502 13858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:34 dist-test-slave-rnrw krb5kdc[13741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307014, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:34.038695 13858 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:23:34.040138 13858 webserver.cc:460] Webserver started at http://127.12.78.1:40639/ using document root <none> and password file <none>
I20260809 20:23:34.040920 13858 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:34.041055 13858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:34.041390 13858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:34.044440 13858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "73cb4f49244243e3a27159c6497f8074"
format_stamp: "Formatted at 2026-08-09 20:23:34 on dist-test-slave-rnrw"
server_key: "84a505da0cf08bb02ddac40c9b40d6c8"
server_key_iv: "cbfd74813c857c6bd4165bb18a8556d4"
server_key_version: "encryptionkey@0"
I20260809 20:23:34.045313 13858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "73cb4f49244243e3a27159c6497f8074"
format_stamp: "Formatted at 2026-08-09 20:23:34 on dist-test-slave-rnrw"
server_key: "84a505da0cf08bb02ddac40c9b40d6c8"
server_key_iv: "cbfd74813c857c6bd4165bb18a8556d4"
server_key_version: "encryptionkey@0"
I20260809 20:23:34.050382 13858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:23:34.053787 13873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:34.055015 13858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:23:34.055202 13858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "73cb4f49244243e3a27159c6497f8074"
format_stamp: "Formatted at 2026-08-09 20:23:34 on dist-test-slave-rnrw"
server_key: "84a505da0cf08bb02ddac40c9b40d6c8"
server_key_iv: "cbfd74813c857c6bd4165bb18a8556d4"
server_key_version: "encryptionkey@0"
I20260809 20:23:34.055433 13858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:34.088982 13858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:34.093734 13858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:34.094055 13858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:34.097012 13858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:34.097121 13858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:34.097283 13858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:34.097368 13858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 20:23:34 dist-test-slave-rnrw krb5kdc[13741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307014, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:34.122807 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:34.097637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:42561 (local address 127.12.78.62:38475)
0809 20:23:34.100138 (+ 2501us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:34.100146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:34.100170 (+ 24us) server_negotiation.cc:407] Connection header received
0809 20:23:34.101938 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:34.101947 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:34.102063 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:34.102183 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:34.104400 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.105838 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.108358 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.109155 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.109351 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:34.115569 (+ 6218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:34.115580 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:34.115588 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:34.115643 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:34.119156 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:34.120110 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:34.120118 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:34.120124 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:34.120302 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:34.120920 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:34.120926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:34.120931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:34.121353 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:34.121502 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:34.122221 (+ 719us) server_negotiation.cc:299] Negotiation successful
0809 20:23:34.122441 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2348,"thread_start_us":2236,"threads_started":1}
I20260809 20:23:34.123816 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:34.098322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:38475 (local address 127.12.78.1:42561)
0809 20:23:34.099423 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:34.099487 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:34.101622 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:34.102456 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:34.102470 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:34.103233 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:34.104130 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:34.104166 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.106009 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.106016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:34.108201 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:34.108213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.109815 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.109830 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:34.110243 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:34.111271 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:34.111311 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:34.115214 (+ 3903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:34.119290 (+ 4076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:34.119300 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:34.119320 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:34.119985 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:34.120460 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:34.120470 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:34.120473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:34.120795 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:34.121488 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:34.121495 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:34.121859 (+ 364us) client_negotiation.cc:769] Sending connection context
0809 20:23:34.122117 (+ 258us) client_negotiation.cc:240] Negotiation successful
0809 20:23:34.122642 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":449,"threads_started":1}
I20260809 20:23:34.148427 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:34.131321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:58439 (local address 127.12.78.62:38475)
0809 20:23:34.131649 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:34.131656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:34.132281 (+ 625us) server_negotiation.cc:407] Connection header received
0809 20:23:34.132487 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:34.132492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:34.132594 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:34.132689 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:34.134346 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.135021 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.137570 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.138055 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.138203 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:34.140976 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:34.140986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:34.140993 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:34.141032 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:34.144881 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:34.145985 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:34.145993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:34.145996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:34.146181 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:34.146629 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:34.146635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:34.146642 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:34.147077 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:34.147266 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:34.147796 (+ 530us) server_negotiation.cc:299] Negotiation successful
0809 20:23:34.148031 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260809 20:23:34.148583 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:34.131344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:38475 (local address 127.12.78.1:58439)
0809 20:23:34.132144 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:34.132171 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:34.132343 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:34.132762 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:34.132770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:34.133568 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:34.134169 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:34.134187 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.135174 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.135180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:34.137385 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:34.137409 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.138276 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.138286 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:34.138524 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:34.139235 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:34.139268 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:34.140702 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:34.145073 (+ 4371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:34.145077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:34.145081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:34.145836 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:34.146301 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:34.146306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:34.146310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:34.146498 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:34.147246 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:34.147254 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:34.147509 (+ 255us) client_negotiation.cc:769] Sending connection context
0809 20:23:34.147689 (+ 180us) client_negotiation.cc:240] Negotiation successful
0809 20:23:34.148068 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":427,"threads_started":1}
I20260809 20:23:34.157238 13858 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:40589
I20260809 20:23:34.157280 14004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:40589 every 8 connection(s)
I20260809 20:23:34.158516 13858 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:23:34.160993 14005 heartbeater.cc:347] Connected to a master server at 127.12.78.62:38475
I20260809 20:23:34.161252 14005 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:34.161989 14005 heartbeater.cc:511] Master 127.12.78.62:38475 requested a full tablet report, sending...
I20260809 20:23:34.162449 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 13858
I20260809 20:23:34.162671 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:23:34.163509 12600 external_mini_cluster.cc:1428] Setting key ae8f2ff026daa19a07f0ee26b16afce2
I20260809 20:23:34.164448 13787 ts_manager.cc:194] Registered new tserver with Master: 73cb4f49244243e3a27159c6497f8074 (127.12.78.1:40589)
I20260809 20:23:34.166247 13787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:42561
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:23:34.207304 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:38475
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:34.545315 14009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:34.545630 14009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:34.545801 14009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:34.553051 14009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:34.553206 14009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:34.553349 14009 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 20:23:34.553488 14009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:23:34.563526 14009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.78.62:38475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:34.564965 14009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:34.566341 14009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:34.579572 14014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:34.579613 14015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:34.580096 14017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:34.580734 14009 server_base.cc:1034] running on GCE node
I20260809 20:23:34.581372 14009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:34.582252 14009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:34.583461 14009 hybrid_clock.cc:648] HybridClock initialized: now 1786307014583393 us; error 42 us; skew 500 ppm
I20260809 20:23:34.583722 14009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:34 dist-test-slave-rnrw krb5kdc[13741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307014, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:34.586961 14009 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:23:34.588326 14009 webserver.cc:460] Webserver started at http://127.12.78.2:40993/ using document root <none> and password file <none>
I20260809 20:23:34.589041 14009 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:34.589129 14009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:34.589395 14009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:34.592167 14009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "00c00c7560664040b2299167c7570374"
format_stamp: "Formatted at 2026-08-09 20:23:34 on dist-test-slave-rnrw"
server_key: "077dc0db96920c9f3f7e609071bc7af2"
server_key_iv: "a0b6fb3b0d1cc62d01a82745cc689ee2"
server_key_version: "encryptionkey@0"
I20260809 20:23:34.593001 14009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "00c00c7560664040b2299167c7570374"
format_stamp: "Formatted at 2026-08-09 20:23:34 on dist-test-slave-rnrw"
server_key: "077dc0db96920c9f3f7e609071bc7af2"
server_key_iv: "a0b6fb3b0d1cc62d01a82745cc689ee2"
server_key_version: "encryptionkey@0"
I20260809 20:23:34.598088 14009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 20:23:34.601436 14024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:34.602715 14009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:23:34.602846 14009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "00c00c7560664040b2299167c7570374"
format_stamp: "Formatted at 2026-08-09 20:23:34 on dist-test-slave-rnrw"
server_key: "077dc0db96920c9f3f7e609071bc7af2"
server_key_iv: "a0b6fb3b0d1cc62d01a82745cc689ee2"
server_key_version: "encryptionkey@0"
I20260809 20:23:34.603020 14009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:34.616490 14009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:34.620793 14009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:34.621013 14009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:34.623514 14009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:34.623627 14009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:34.623813 14009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:34.623894 14009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 20:23:34 dist-test-slave-rnrw krb5kdc[13741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307014, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:34.648666 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:34.624263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:33549 (local address 127.12.78.62:38475)
0809 20:23:34.624500 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:34.624506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:34.625999 (+ 1493us) server_negotiation.cc:407] Connection header received
0809 20:23:34.629117 (+ 3118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:34.629123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:34.629222 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:34.629315 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:34.631992 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.633005 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.635596 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.636148 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.636292 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:34.641057 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:34.641067 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:34.641075 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:34.641133 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:34.644646 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:34.645647 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:34.645655 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:34.645659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:34.645846 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:34.646367 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:34.646376 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:34.646382 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:34.646878 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:34.647091 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:34.647790 (+ 699us) server_negotiation.cc:299] Negotiation successful
0809 20:23:34.648102 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 20:23:34.649317 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:34.624823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:38475 (local address 127.12.78.2:33549)
0809 20:23:34.625803 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:34.625862 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:34.628781 (+ 2919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:34.629636 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:34.629648 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:34.630595 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:34.631699 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:34.631728 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.633262 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.633273 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:34.635437 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:34.635449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.636407 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.636417 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:34.636729 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:34.637558 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:34.637582 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:34.640707 (+ 3125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:34.644816 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:34.644823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:34.644834 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:34.645513 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:34.646017 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:34.646022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:34.646026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:34.646237 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:34.647034 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:34.647040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:34.647370 (+ 330us) client_negotiation.cc:769] Sending connection context
0809 20:23:34.647627 (+ 257us) client_negotiation.cc:240] Negotiation successful
0809 20:23:34.648021 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":426,"threads_started":1}
I20260809 20:23:34.675390 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:34.660431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:48563 (local address 127.12.78.62:38475)
0809 20:23:34.660654 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:34.660661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:34.660909 (+ 248us) server_negotiation.cc:407] Connection header received
0809 20:23:34.661202 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:34.661208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:34.661286 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:34.661392 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:34.662443 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.663200 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.664534 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.665134 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.665273 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:34.668164 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:34.668174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:34.668182 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:34.668235 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:34.671930 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:34.672732 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:34.672741 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:34.672748 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:34.672939 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:34.673424 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:34.673432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:34.673435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:34.673954 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:34.674117 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:34.674631 (+ 514us) server_negotiation.cc:299] Negotiation successful
0809 20:23:34.674931 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 20:23:34.675580 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:34.659788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:38475 (local address 127.12.78.2:48563)
0809 20:23:34.660798 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:34.660818 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:34.661037 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:34.661420 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:34.661425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:34.661982 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:34.662292 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:34.662301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.663350 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.663356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:34.664386 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:34.664398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:34.665513 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:34.665521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:34.665732 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:34.666544 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:34.666575 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:34.667918 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:34.672087 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:34.672092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:34.672096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:34.672517 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:34.673096 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:34.673104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:34.673110 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:34.673301 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:34.674089 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:34.674092 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:34.674288 (+ 196us) client_negotiation.cc:769] Sending connection context
0809 20:23:34.674501 (+ 213us) client_negotiation.cc:240] Negotiation successful
0809 20:23:34.674979 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":868,"threads_started":1}
I20260809 20:23:34.684334 14009 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:32969
I20260809 20:23:34.684367 14148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:32969 every 8 connection(s)
I20260809 20:23:34.685640 14009 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:23:34.687073 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 14009
I20260809 20:23:34.687302 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:23:34.687752 14149 heartbeater.cc:347] Connected to a master server at 127.12.78.62:38475
I20260809 20:23:34.687865 12600 external_mini_cluster.cc:1428] Setting key 2d57eaf1bcb826b515544aba5b9650d8
I20260809 20:23:34.687978 14149 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:34.688699 14149 heartbeater.cc:511] Master 127.12.78.62:38475 requested a full tablet report, sending...
I20260809 20:23:34.689986 13787 ts_manager.cc:194] Registered new tserver with Master: 00c00c7560664040b2299167c7570374 (127.12.78.2:32969)
I20260809 20:23:34.690793 13787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:33549
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:23:34.731546 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:38475
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:35.072944 14153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:35.073237 14153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:35.073335 14153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:35.080629 14153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:35.080744 14153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:35.080827 14153 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 20:23:35.080909 14153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:23:35.090867 14153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.78.62:38475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:35.092411 14153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:35.093814 14153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:35.107491 14158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:35.107537 14161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:35.107568 14159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:35.108911 14153 server_base.cc:1034] running on GCE node
I20260809 20:23:35.109543 14153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:35.110524 14153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:35.111752 14153 hybrid_clock.cc:648] HybridClock initialized: now 1786307015111683 us; error 42 us; skew 500 ppm
I20260809 20:23:35.112066 14153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:35 dist-test-slave-rnrw krb5kdc[13741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307015, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:35.116479 14153 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:23:35.118098 14153 webserver.cc:460] Webserver started at http://127.12.78.3:37837/ using document root <none> and password file <none>
I20260809 20:23:35.118799 14153 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:35.118891 14153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:35.119169 14153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:35.122347 14153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "fa9186a1feba4fb7a9f15c18d03ca226"
format_stamp: "Formatted at 2026-08-09 20:23:35 on dist-test-slave-rnrw"
server_key: "d1c7315a1d02e10b404fb9da806d98a3"
server_key_iv: "b33c18f7dadbad28469f1e298a9e00b8"
server_key_version: "encryptionkey@0"
I20260809 20:23:35.123124 14153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "fa9186a1feba4fb7a9f15c18d03ca226"
format_stamp: "Formatted at 2026-08-09 20:23:35 on dist-test-slave-rnrw"
server_key: "d1c7315a1d02e10b404fb9da806d98a3"
server_key_iv: "b33c18f7dadbad28469f1e298a9e00b8"
server_key_version: "encryptionkey@0"
I20260809 20:23:35.128506 14153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 20:23:35.132123 14168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:35.133599 14153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:23:35.133796 14153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "fa9186a1feba4fb7a9f15c18d03ca226"
format_stamp: "Formatted at 2026-08-09 20:23:35 on dist-test-slave-rnrw"
server_key: "d1c7315a1d02e10b404fb9da806d98a3"
server_key_iv: "b33c18f7dadbad28469f1e298a9e00b8"
server_key_version: "encryptionkey@0"
I20260809 20:23:35.134037 14153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:35.165022 14153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:35.169371 14005 heartbeater.cc:503] Master 127.12.78.62:38475 was elected leader, sending a full tablet report...
I20260809 20:23:35.169728 14153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:35.170085 14153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:35.173178 14153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:35.173300 14153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:35.173471 14153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:35.173550 14153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 20:23:35 dist-test-slave-rnrw krb5kdc[13741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307015, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:35.204836 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.174650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:38475 (local address 127.12.78.3:40369)
0809 20:23:35.175644 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:35.175712 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:35.179112 (+ 3400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:35.180188 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:35.180209 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:35.181404 (+ 1195us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:35.182489 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.182524 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.183951 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.183960 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.187028 (+ 3068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.187063 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.188232 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.188243 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.188683 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.189782 (+ 1099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:35.189823 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:35.193413 (+ 3590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:35.197668 (+ 4255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:35.197681 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:35.197703 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:35.198550 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:35.199123 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:35.199132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:35.199138 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:35.199543 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:35.200517 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:35.200531 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:35.201168 (+ 637us) client_negotiation.cc:769] Sending connection context
0809 20:23:35.201735 (+ 567us) client_negotiation.cc:240] Negotiation successful
0809 20:23:35.202545 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":365,"threads_started":1}
I20260809 20:23:35.205432 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.174042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:40369 (local address 127.12.78.62:38475)
0809 20:23:35.174249 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:35.174256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:35.175862 (+ 1606us) server_negotiation.cc:407] Connection header received
0809 20:23:35.179533 (+ 3671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:35.179543 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:35.179668 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:35.179801 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:35.182749 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.183741 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.187279 (+ 3538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.187899 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.188064 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.193782 (+ 5718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:35.193791 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:35.193797 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:35.193848 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:35.197487 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:35.198731 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:35.198739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:35.198745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:35.198949 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:35.199747 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:35.199756 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:35.199762 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:35.200205 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:35.200408 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:35.204622 (+ 4214us) server_negotiation.cc:299] Negotiation successful
0809 20:23:35.204929 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 20:23:35.233346 14267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.211848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:38475 (local address 127.12.78.3:52723)
0809 20:23:35.217484 (+ 5636us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:35.217515 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:35.217742 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:35.218126 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:35.218134 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:35.218975 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:35.219503 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.219521 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.220615 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.220624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.222620 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.222641 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.223944 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.223954 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.224084 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.224836 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:35.224854 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:35.226259 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:35.229862 (+ 3603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:35.229868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:35.229872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:35.230749 (+ 877us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:35.231209 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:35.231217 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:35.231229 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:35.231474 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:35.232160 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:35.232163 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:35.232323 (+ 160us) client_negotiation.cc:769] Sending connection context
0809 20:23:35.232463 (+ 140us) client_negotiation.cc:240] Negotiation successful
0809 20:23:35.232925 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5384,"thread_start_us":5154,"threads_started":1}
I20260809 20:23:35.233547 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.211598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:52723 (local address 127.12.78.62:38475)
0809 20:23:35.211765 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:35.211769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:35.217684 (+ 5915us) server_negotiation.cc:407] Connection header received
0809 20:23:35.217880 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:35.217884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:35.217955 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:35.218047 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:35.219700 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.220401 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.222871 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.223694 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.223846 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.226489 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:35.226495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:35.226498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:35.226532 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:35.229724 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:35.230916 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:35.230920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:35.230922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:35.231052 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:35.231667 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:35.231673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:35.231676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:35.232037 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:35.232205 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:35.232757 (+ 552us) server_negotiation.cc:299] Negotiation successful
0809 20:23:35.233034 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 20:23:35.237790 14153 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:34137
I20260809 20:23:35.237854 14296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:34137 every 8 connection(s)
I20260809 20:23:35.239223 14153 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:23:35.241523 14297 heartbeater.cc:347] Connected to a master server at 127.12.78.62:38475
I20260809 20:23:35.241796 14297 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:35.242357 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 14153
I20260809 20:23:35.242465 14297 heartbeater.cc:511] Master 127.12.78.62:38475 requested a full tablet report, sending...
I20260809 20:23:35.242579 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:23:35.243054 12600 external_mini_cluster.cc:1428] Setting key fbed1b703728cb216a6593f0aa47b289
I20260809 20:23:35.244053 13787 ts_manager.cc:194] Registered new tserver with Master: fa9186a1feba4fb7a9f15c18d03ca226 (127.12.78.3:34137)
I20260809 20:23:35.245159 13787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:40369
I20260809 20:23:35.246323 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 20:23:35 dist-test-slave-rnrw krb5kdc[13741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 20:23:35 dist-test-slave-rnrw krb5kdc[13741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:35.298144 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.284143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.12.78.62:38475)
0809 20:23:35.284403 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:35.284415 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:35.284539 (+ 124us) server_negotiation.cc:407] Connection header received
0809 20:23:35.284796 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:35.284800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:35.284866 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:35.284956 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:35.286465 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.287248 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.288537 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.289017 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.289115 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.293103 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:35.293108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:35.293111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:35.293145 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:35.295641 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:35.296250 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:35.296253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:35.296255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:35.296381 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:35.296799 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:35.296802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:35.296805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:35.297084 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:35.297300 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:35.297586 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 20:23:35.297761 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260809 20:23:35.302380 13786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:23:35.309773 13786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:23:35.343585 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.322796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:40589 (local address 127.0.0.1:59962)
0809 20:23:35.334894 (+ 12098us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:35.334938 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:35.335142 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:35.336166 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:35.336174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:35.336202 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:35.336676 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.336691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.338503 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.338508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.340730 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.340743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.342440 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.342446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.342658 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.342710 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 20:23:35.342848 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 20:23:35.343039 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11935,"mutex_wait_us":22,"spinlock_wait_cycles":3328,"thread_start_us":13111,"threads_started":1}
I20260809 20:23:35.344208 14311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.323114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:32969 (local address 127.0.0.1:43864)
0809 20:23:35.335662 (+ 12548us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:35.335680 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:35.335804 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:35.336498 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:35.336503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:35.336535 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:35.336989 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.337002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.338870 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.338877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.341261 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.341275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.343435 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.343443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.343620 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.343655 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 20:23:35.343733 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 20:23:35.343856 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12431,"thread_start_us":11461,"threads_started":1}
I20260809 20:23:35.344847 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.324192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.12.78.2:32969)
0809 20:23:35.333068 (+ 8876us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:35.333079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:35.335829 (+ 2750us) server_negotiation.cc:407] Connection header received
0809 20:23:35.335971 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:35.335981 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:35.336256 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:35.336497 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:35.337198 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.338689 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.341437 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.342760 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.342912 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.343105 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:35.344404 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0809 20:23:35.344635 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8678,"thread_start_us":8244,"threads_started":1}
I20260809 20:23:35.344910 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.323700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.12.78.1:40589)
0809 20:23:35.332274 (+ 8574us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:35.332287 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:35.335401 (+ 3114us) server_negotiation.cc:407] Connection header received
0809 20:23:35.335525 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:35.335535 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:35.335820 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:35.336006 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:35.336890 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.338326 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.340882 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.342219 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.342363 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.342577 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:35.342928 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 20:23:35.344606 (+ 1678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8322,"thread_start_us":8142,"threads_started":1}
I20260809 20:23:35.346041 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.322518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:34137 (local address 127.0.0.1:34664)
0809 20:23:35.334920 (+ 12402us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:35.334948 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:35.335146 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:35.338330 (+ 3184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:35.338335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:35.338359 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:35.339567 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.339583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.341593 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.341601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.343321 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.343341 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.345234 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.345240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.345417 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.345449 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 20:23:35.345545 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 20:23:35.345667 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11982,"mutex_wait_us":1,"spinlock_wait_cycles":5376,"thread_start_us":12820,"threads_started":1}
I20260809 20:23:35.346068 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.322809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34664 (local address 127.12.78.3:34137)
0809 20:23:35.337445 (+ 14636us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:35.337454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:35.337485 (+ 31us) server_negotiation.cc:407] Connection header received
0809 20:23:35.337571 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:35.337577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:35.337800 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:35.338021 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:35.339822 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.341460 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.343560 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.345043 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.345196 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.345429 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:35.345580 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 20:23:35.345740 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14454,"thread_start_us":14243,"threads_started":1}
I20260809 20:23:35.348464 14077 tablet_service.cc:1467] Processing CreateTablet for tablet eec69fa2542c4f3bbe6425738c6b33de (DEFAULT_TABLE table=test-table [id=7e72ed47fdc8457ab0384cd2141813c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:35.348841 13928 tablet_service.cc:1467] Processing CreateTablet for tablet eec69fa2542c4f3bbe6425738c6b33de (DEFAULT_TABLE table=test-table [id=7e72ed47fdc8457ab0384cd2141813c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:35.350069 14077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec69fa2542c4f3bbe6425738c6b33de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:35.349650 14221 tablet_service.cc:1467] Processing CreateTablet for tablet eec69fa2542c4f3bbe6425738c6b33de (DEFAULT_TABLE table=test-table [id=7e72ed47fdc8457ab0384cd2141813c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:35.350301 13928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec69fa2542c4f3bbe6425738c6b33de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:35.351100 14221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec69fa2542c4f3bbe6425738c6b33de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:35.362406 14317 tablet_bootstrap.cc:492] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374: Bootstrap starting.
I20260809 20:23:35.365554 14318 tablet_bootstrap.cc:492] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074: Bootstrap starting.
I20260809 20:23:35.366426 14319 tablet_bootstrap.cc:492] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226: Bootstrap starting.
I20260809 20:23:35.367340 14317 tablet_bootstrap.cc:654] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:35.368858 14317 log.cc:824] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:35.372342 14319 tablet_bootstrap.cc:654] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:35.372483 14317 tablet_bootstrap.cc:492] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374: No bootstrap required, opened a new log
I20260809 20:23:35.372807 14317 ts_tablet_manager.cc:1397] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260809 20:23:35.373787 14319 log.cc:824] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:35.373965 14318 tablet_bootstrap.cc:654] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:35.375252 14318 log.cc:824] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:35.377068 14319 tablet_bootstrap.cc:492] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226: No bootstrap required, opened a new log
I20260809 20:23:35.377348 14319 ts_tablet_manager.cc:1397] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260809 20:23:35.378602 14318 tablet_bootstrap.cc:492] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074: No bootstrap required, opened a new log
I20260809 20:23:35.378895 14318 ts_tablet_manager.cc:1397] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20260809 20:23:35.385571 14317 raft_consensus.cc:348] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } }
I20260809 20:23:35.386260 14317 raft_consensus.cc:374] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:35.386472 14317 raft_consensus.cc:729] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c00c7560664040b2299167c7570374, State: Initialized, Role: FOLLOWER
I20260809 20:23:35.387390 14317 consensus_queue.cc:260] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } }
I20260809 20:23:35.388837 14149 heartbeater.cc:503] Master 127.12.78.62:38475 was elected leader, sending a full tablet report...
I20260809 20:23:35.389010 14318 raft_consensus.cc:348] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } }
I20260809 20:23:35.389523 14318 raft_consensus.cc:374] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:35.389622 14317 ts_tablet_manager.cc:1428] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260809 20:23:35.389688 14318 raft_consensus.cc:729] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73cb4f49244243e3a27159c6497f8074, State: Initialized, Role: FOLLOWER
I20260809 20:23:35.390408 14318 consensus_queue.cc:260] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } }
I20260809 20:23:35.392496 14319 raft_consensus.cc:348] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } }
I20260809 20:23:35.393222 14318 ts_tablet_manager.cc:1428] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074: Time spent starting tablet: real 0.014s user 0.005s sys 0.010s
I20260809 20:23:35.393201 14319 raft_consensus.cc:374] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:35.393442 14319 raft_consensus.cc:729] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9186a1feba4fb7a9f15c18d03ca226, State: Initialized, Role: FOLLOWER
I20260809 20:23:35.394099 14319 consensus_queue.cc:260] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } }
I20260809 20:23:35.396032 14297 heartbeater.cc:503] Master 127.12.78.62:38475 was elected leader, sending a full tablet report...
I20260809 20:23:35.396648 14319 ts_tablet_manager.cc:1428] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226: Time spent starting tablet: real 0.019s user 0.008s sys 0.010s
W20260809 20:23:35.411123 14006 tablet.cc:2369] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:23:35.438692 14150 tablet.cc:2369] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:23:35.489056 14324 raft_consensus.cc:482] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:23:35.489398 14324 raft_consensus.cc:504] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } }
W20260809 20:23:35.490993 14298 tablet.cc:2369] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:23:35.491637 14324 leader_election.cc:288] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa9186a1feba4fb7a9f15c18d03ca226 (127.12.78.3:34137), 00c00c7560664040b2299167c7570374 (127.12.78.2:32969)
I20260809 20:23:35.500265 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.492583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:57997 (local address 127.12.78.2:32969)
0809 20:23:35.492858 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:35.492866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:35.493637 (+ 771us) server_negotiation.cc:407] Connection header received
0809 20:23:35.493874 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:35.493882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:35.493948 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:35.494054 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:35.494843 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.495695 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.498199 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.499169 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.499336 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.499453 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:35.499792 (+ 339us) server_negotiation.cc:299] Negotiation successful
0809 20:23:35.499928 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 20:23:35.500509 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.491890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:32969 (local address 127.12.78.1:57997)
0809 20:23:35.493508 (+ 1618us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:35.493525 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:35.493730 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:35.494119 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:35.494124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:35.494159 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:35.494694 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.494704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.495917 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.495925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.497989 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.498002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.499398 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.499403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.499623 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.499648 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 20:23:35.499761 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 20:23:35.499895 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1465,"thread_start_us":1642,"threads_started":1}
I20260809 20:23:35.501384 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.492499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:32895 (local address 127.12.78.3:34137)
0809 20:23:35.492710 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:35.492717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:35.494040 (+ 1323us) server_negotiation.cc:407] Connection header received
0809 20:23:35.494238 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:35.494243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:35.494313 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:35.494413 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:35.495454 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.496379 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.498847 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.499922 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.500086 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.500280 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:35.500826 (+ 546us) server_negotiation.cc:299] Negotiation successful
0809 20:23:35.500963 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 20:23:35.502175 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.491928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:34137 (local address 127.12.78.1:32895)
0809 20:23:35.493878 (+ 1950us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:35.493893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:35.494079 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:35.494439 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:35.494443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:35.494468 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:35.495266 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.495277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.496683 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.496687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.498644 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.498657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.500317 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.500325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.500631 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.500664 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 20:23:35.501541 (+ 877us) client_negotiation.cc:240] Negotiation successful
0809 20:23:35.501718 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1822,"thread_start_us":1098,"threads_started":1}
I20260809 20:23:35.502518 14103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec69fa2542c4f3bbe6425738c6b33de" candidate_uuid: "73cb4f49244243e3a27159c6497f8074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c00c7560664040b2299167c7570374" is_pre_election: true
I20260809 20:23:35.503217 14103 raft_consensus.cc:2454] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73cb4f49244243e3a27159c6497f8074 in term 0.
I20260809 20:23:35.503409 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec69fa2542c4f3bbe6425738c6b33de" candidate_uuid: "73cb4f49244243e3a27159c6497f8074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" is_pre_election: true
I20260809 20:23:35.504248 13877 leader_election.cc:302] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c00c7560664040b2299167c7570374, 73cb4f49244243e3a27159c6497f8074; no voters:
I20260809 20:23:35.504181 14243 raft_consensus.cc:2454] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73cb4f49244243e3a27159c6497f8074 in term 0.
I20260809 20:23:35.504711 14324 raft_consensus.cc:2781] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:23:35.504957 14324 raft_consensus.cc:482] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:23:35.505108 14324 raft_consensus.cc:3037] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:35.508217 14324 raft_consensus.cc:504] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } }
I20260809 20:23:35.508915 14324 leader_election.cc:288] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [CANDIDATE]: Term 1 election: Requested vote from peers fa9186a1feba4fb7a9f15c18d03ca226 (127.12.78.3:34137), 00c00c7560664040b2299167c7570374 (127.12.78.2:32969)
I20260809 20:23:35.509479 14103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec69fa2542c4f3bbe6425738c6b33de" candidate_uuid: "73cb4f49244243e3a27159c6497f8074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c00c7560664040b2299167c7570374"
I20260809 20:23:35.509600 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec69fa2542c4f3bbe6425738c6b33de" candidate_uuid: "73cb4f49244243e3a27159c6497f8074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9186a1feba4fb7a9f15c18d03ca226"
I20260809 20:23:35.509727 14103 raft_consensus.cc:3037] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:35.509851 14243 raft_consensus.cc:3037] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:35.514472 14103 raft_consensus.cc:2454] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73cb4f49244243e3a27159c6497f8074 in term 1.
I20260809 20:23:35.514443 14243 raft_consensus.cc:2454] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73cb4f49244243e3a27159c6497f8074 in term 1.
I20260809 20:23:35.514971 13876 leader_election.cc:302] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73cb4f49244243e3a27159c6497f8074, fa9186a1feba4fb7a9f15c18d03ca226; no voters:
I20260809 20:23:35.515393 14324 raft_consensus.cc:2781] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:35.516019 14324 raft_consensus.cc:686] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [term 1 LEADER]: Becoming Leader. State: Replica: 73cb4f49244243e3a27159c6497f8074, State: Running, Role: LEADER
I20260809 20:23:35.516610 14324 consensus_queue.cc:237] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } }
I20260809 20:23:35.522569 13787 catalog_manager.cc:5495] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73cb4f49244243e3a27159c6497f8074 (127.12.78.1). New cstate: current_term: 1 leader_uuid: "73cb4f49244243e3a27159c6497f8074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } health_report { overall_health: UNKNOWN } } }
I20260809 20:23:35.585139 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.578929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.12.78.1:40589)
0809 20:23:35.579203 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:35.579210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:35.579233 (+ 23us) server_negotiation.cc:407] Connection header received
0809 20:23:35.579552 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:35.579556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:35.579637 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:35.579827 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:23:35.580233 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.581046 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.582573 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.583042 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.583182 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.583874 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:23:35.584455 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:23:35.584643 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:35.584757 (+ 114us) server_negotiation.cc:299] Negotiation successful
0809 20:23:35.584842 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 20:23:35.595017 14103 raft_consensus.cc:1261] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374 [term 1 FOLLOWER]: Refusing update from remote peer 73cb4f49244243e3a27159c6497f8074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:35.594981 14243 raft_consensus.cc:1261] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226 [term 1 FOLLOWER]: Refusing update from remote peer 73cb4f49244243e3a27159c6497f8074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:35.596340 14328 consensus_queue.cc:1035] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:35.596804 14324 consensus_queue.cc:1035] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:35.612447 14332 mvcc.cc:205] Tried to move back new op lower bound from 7316713535864754176 to 7316713535561756672. Current Snapshot: MvccSnapshot[applied={T|T < 7316713535864754176}]
I20260809 20:23:35.613909 14331 mvcc.cc:205] Tried to move back new op lower bound from 7316713535864754176 to 7316713535561756672. Current Snapshot: MvccSnapshot[applied={T|T < 7316713535864754176}]
I20260809 20:23:35.636196 13787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:23:35.636590 13787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:23:35.641263 13787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c: Sending DeleteTablet for 3 replicas of tablet eec69fa2542c4f3bbe6425738c6b33de
I20260809 20:23:35.642414 13928 tablet_service.cc:1514] Processing DeleteTablet for tablet eec69fa2542c4f3bbe6425738c6b33de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:35 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:59962
I20260809 20:23:35.642421 14221 tablet_service.cc:1514] Processing DeleteTablet for tablet eec69fa2542c4f3bbe6425738c6b33de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:35 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:34664
I20260809 20:23:35.642490 14077 tablet_service.cc:1514] Processing DeleteTablet for tablet eec69fa2542c4f3bbe6425738c6b33de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:35 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:43864
I20260809 20:23:35.644690 13786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:23:35.645717 13786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:23:35.649966 14343 tablet_replica.cc:330] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374: stopping tablet replica
I20260809 20:23:35.650558 14343 raft_consensus.cc:2229] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:23:35.651445 14343 raft_consensus.cc:2258] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:35.655400 13927 tablet_service.cc:1467] Processing CreateTablet for tablet 46eab50d90fd41c291582f78077ab53d (DEFAULT_TABLE table=test-table [id=b70e38ad30ec4e28bf23b162988ff828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:35.656004 14077 tablet_service.cc:1467] Processing CreateTablet for tablet 46eab50d90fd41c291582f78077ab53d (DEFAULT_TABLE table=test-table [id=b70e38ad30ec4e28bf23b162988ff828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:35.656226 13927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46eab50d90fd41c291582f78077ab53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:35.656484 14077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46eab50d90fd41c291582f78077ab53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:35.656914 14220 tablet_service.cc:1467] Processing CreateTablet for tablet 46eab50d90fd41c291582f78077ab53d (DEFAULT_TABLE table=test-table [id=b70e38ad30ec4e28bf23b162988ff828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:35.657483 14342 tablet_replica.cc:330] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074: stopping tablet replica
I20260809 20:23:35.657681 14220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46eab50d90fd41c291582f78077ab53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:35.664908 14317 tablet_bootstrap.cc:492] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374: Bootstrap starting.
I20260809 20:23:35.665488 14318 tablet_bootstrap.cc:492] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074: Bootstrap starting.
I20260809 20:23:35.667771 14317 tablet_bootstrap.cc:654] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:35.667868 14318 tablet_bootstrap.cc:654] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:35.667951 14343 ts_tablet_manager.cc:1905] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:35.672232 14341 tablet_replica.cc:330] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226: stopping tablet replica
I20260809 20:23:35.673117 14341 raft_consensus.cc:2229] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:23:35.674412 14342 raft_consensus.cc:2229] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:23:35.675046 14319 tablet_bootstrap.cc:492] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226: Bootstrap starting.
I20260809 20:23:35.678246 14319 tablet_bootstrap.cc:654] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:35.679520 14317 tablet_bootstrap.cc:492] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374: No bootstrap required, opened a new log
I20260809 20:23:35.679769 14317 ts_tablet_manager.cc:1397] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260809 20:23:35.681288 14317 raft_consensus.cc:348] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.681658 14317 raft_consensus.cc:374] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:35.681864 14317 raft_consensus.cc:729] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c00c7560664040b2299167c7570374, State: Initialized, Role: FOLLOWER
I20260809 20:23:35.682406 14317 consensus_queue.cc:260] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.683494 14343 ts_tablet_manager.cc:1918] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:23:35.683745 14343 log.cc:1196] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/eec69fa2542c4f3bbe6425738c6b33de
I20260809 20:23:35.685724 14323 raft_consensus.cc:482] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:23:35.685984 14323 raft_consensus.cc:504] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.684576 14317 ts_tablet_manager.cc:1428] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 20:23:35.687462 14341 raft_consensus.cc:2258] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:35.688937 14342 raft_consensus.cc:2258] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:35.689963 14343 ts_tablet_manager.cc:1939] T eec69fa2542c4f3bbe6425738c6b33de P 00c00c7560664040b2299167c7570374: Deleting consensus metadata
I20260809 20:23:35.692032 13773 catalog_manager.cc:4841] TS 00c00c7560664040b2299167c7570374 (127.12.78.2:32969): tablet eec69fa2542c4f3bbe6425738c6b33de (table test-table [id=7e72ed47fdc8457ab0384cd2141813c3]) successfully deleted
I20260809 20:23:35.694074 14341 ts_tablet_manager.cc:1905] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:35.699429 14342 ts_tablet_manager.cc:1905] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:35.700321 14323 leader_election.cc:288] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73cb4f49244243e3a27159c6497f8074 (127.12.78.1:40589), fa9186a1feba4fb7a9f15c18d03ca226 (127.12.78.3:34137)
I20260809 20:23:35.703853 14318 tablet_bootstrap.cc:492] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074: No bootstrap required, opened a new log
I20260809 20:23:35.704062 14318 ts_tablet_manager.cc:1397] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.012s
I20260809 20:23:35.705395 14318 raft_consensus.cc:348] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.705775 14318 raft_consensus.cc:374] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:35.705910 14318 raft_consensus.cc:729] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73cb4f49244243e3a27159c6497f8074, State: Initialized, Role: FOLLOWER
I20260809 20:23:35.714895 14318 consensus_queue.cc:260] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.719797 14318 ts_tablet_manager.cc:1428] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20260809 20:23:35.721889 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.691697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:58469 (local address 127.12.78.1:40589)
0809 20:23:35.691901 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:35.691910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:35.709205 (+ 17295us) server_negotiation.cc:407] Connection header received
0809 20:23:35.709420 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:35.709426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:35.709521 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:35.709655 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:35.712097 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.714539 (+ 2442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.718475 (+ 3936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.720007 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.720145 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.720284 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:35.721272 (+ 988us) server_negotiation.cc:299] Negotiation successful
0809 20:23:35.721449 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 20:23:35.723184 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.691039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:40589 (local address 127.12.78.2:58469)
0809 20:23:35.708974 (+ 17935us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:35.708996 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:35.709289 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:35.710015 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:35.710020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:35.710049 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:35.710995 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.711011 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.714680 (+ 3669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.714686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.718204 (+ 3518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.718222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.720741 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.720752 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.721034 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.721101 (+ 67us) client_negotiation.cc:769] Sending connection context
0809 20:23:35.722407 (+ 1306us) client_negotiation.cc:240] Negotiation successful
0809 20:23:35.722630 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17729,"thread_start_us":21785,"threads_started":1}
I20260809 20:23:35.724673 13954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46eab50d90fd41c291582f78077ab53d" candidate_uuid: "00c00c7560664040b2299167c7570374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cb4f49244243e3a27159c6497f8074" is_pre_election: true
I20260809 20:23:35.725819 13954 raft_consensus.cc:2454] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c00c7560664040b2299167c7570374 in term 0.
I20260809 20:23:35.727316 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.693519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:51841 (local address 127.12.78.3:34137)
0809 20:23:35.693685 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:35.693691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:35.710237 (+ 16546us) server_negotiation.cc:407] Connection header received
0809 20:23:35.710498 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:35.710504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:35.710576 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:35.710629 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:35.713268 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.714963 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.723866 (+ 8903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.725823 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.725968 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.726066 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:35.726789 (+ 723us) server_negotiation.cc:299] Negotiation successful
0809 20:23:35.726977 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 20:23:35.727373 14026 leader_election.cc:302] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c00c7560664040b2299167c7570374, 73cb4f49244243e3a27159c6497f8074; no voters:
I20260809 20:23:35.728168 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.692831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:34137 (local address 127.12.78.2:51841)
0809 20:23:35.710026 (+ 17195us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:35.710044 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:35.710344 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:35.710751 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:35.710755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:35.710776 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:35.711896 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.711911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.720623 (+ 8712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.720630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.723108 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.723125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.726340 (+ 3215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.726347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.726572 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.726623 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 20:23:35.727605 (+ 982us) client_negotiation.cc:240] Negotiation successful
0809 20:23:35.727759 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17027,"thread_start_us":15745,"threads_started":1}
I20260809 20:23:35.728204 14323 raft_consensus.cc:2781] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:23:35.728361 14323 raft_consensus.cc:482] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:23:35.728528 14323 raft_consensus.cc:3037] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:35.728828 14319 tablet_bootstrap.cc:492] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226: No bootstrap required, opened a new log
I20260809 20:23:35.729067 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46eab50d90fd41c291582f78077ab53d" candidate_uuid: "00c00c7560664040b2299167c7570374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" is_pre_election: true
I20260809 20:23:35.729350 14319 ts_tablet_manager.cc:1397] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226: Time spent bootstrapping tablet: real 0.054s user 0.011s sys 0.002s
W20260809 20:23:35.730067 14027 leader_election.cc:341] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa9186a1feba4fb7a9f15c18d03ca226 (127.12.78.3:34137): Illegal state: must be running to vote when last-logged opid is not known
I20260809 20:23:35.730581 14319 raft_consensus.cc:348] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.730858 14319 raft_consensus.cc:374] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:35.731051 14319 raft_consensus.cc:729] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9186a1feba4fb7a9f15c18d03ca226, State: Initialized, Role: FOLLOWER
I20260809 20:23:35.731488 14319 consensus_queue.cc:260] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.729115 14341 ts_tablet_manager.cc:1918] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:23:35.731966 14341 log.cc:1196] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/eec69fa2542c4f3bbe6425738c6b33de
I20260809 20:23:35.732656 14341 ts_tablet_manager.cc:1939] T eec69fa2542c4f3bbe6425738c6b33de P fa9186a1feba4fb7a9f15c18d03ca226: Deleting consensus metadata
I20260809 20:23:35.732705 14323 raft_consensus.cc:504] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.734091 13954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46eab50d90fd41c291582f78077ab53d" candidate_uuid: "00c00c7560664040b2299167c7570374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cb4f49244243e3a27159c6497f8074"
I20260809 20:23:35.734344 13954 raft_consensus.cc:3037] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:35.734994 14323 leader_election.cc:288] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [CANDIDATE]: Term 1 election: Requested vote from peers 73cb4f49244243e3a27159c6497f8074 (127.12.78.1:40589), fa9186a1feba4fb7a9f15c18d03ca226 (127.12.78.3:34137)
I20260809 20:23:35.736187 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46eab50d90fd41c291582f78077ab53d" candidate_uuid: "00c00c7560664040b2299167c7570374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9186a1feba4fb7a9f15c18d03ca226"
I20260809 20:23:35.736439 14243 raft_consensus.cc:3037] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:35.740288 14243 raft_consensus.cc:2454] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c00c7560664040b2299167c7570374 in term 1.
I20260809 20:23:35.741132 14027 leader_election.cc:302] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c00c7560664040b2299167c7570374, fa9186a1feba4fb7a9f15c18d03ca226; no voters:
I20260809 20:23:35.741580 14323 raft_consensus.cc:2781] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:35.743726 14342 ts_tablet_manager.cc:1918] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:23:35.744017 14342 log.cc:1196] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/eec69fa2542c4f3bbe6425738c6b33de
I20260809 20:23:35.744310 13954 raft_consensus.cc:2454] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c00c7560664040b2299167c7570374 in term 1.
I20260809 20:23:35.745152 14342 ts_tablet_manager.cc:1939] T eec69fa2542c4f3bbe6425738c6b33de P 73cb4f49244243e3a27159c6497f8074: Deleting consensus metadata
I20260809 20:23:35.747591 13771 catalog_manager.cc:4841] TS 73cb4f49244243e3a27159c6497f8074 (127.12.78.1:40589): tablet eec69fa2542c4f3bbe6425738c6b33de (table test-table [id=7e72ed47fdc8457ab0384cd2141813c3]) successfully deleted
I20260809 20:23:35.747800 14319 ts_tablet_manager.cc:1428] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226: Time spent starting tablet: real 0.018s user 0.002s sys 0.001s
I20260809 20:23:35.751542 13772 catalog_manager.cc:4841] TS fa9186a1feba4fb7a9f15c18d03ca226 (127.12.78.3:34137): tablet eec69fa2542c4f3bbe6425738c6b33de (table test-table [id=7e72ed47fdc8457ab0384cd2141813c3]) successfully deleted
I20260809 20:23:35.752614 14323 raft_consensus.cc:686] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [term 1 LEADER]: Becoming Leader. State: Replica: 00c00c7560664040b2299167c7570374, State: Running, Role: LEADER
I20260809 20:23:35.753182 14323 consensus_queue.cc:237] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.757239 13786 catalog_manager.cc:5495] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c00c7560664040b2299167c7570374 (127.12.78.2). New cstate: current_term: 1 leader_uuid: "00c00c7560664040b2299167c7570374" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } health_report { overall_health: UNKNOWN } } }
Aug 09 20:23:35 dist-test-slave-rnrw krb5kdc[13741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307013, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:35.827183 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.807191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:38475 (local address 127.0.0.1:36586)
0809 20:23:35.809181 (+ 1990us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:35.809205 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:35.809413 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:35.809802 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:35.809805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:35.810881 (+ 1076us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:35.811416 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.811427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.813634 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.813638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.814896 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:35.814908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.816108 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.816113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:35.816321 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.817192 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:35.817216 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:35.820095 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:35.824369 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:35.824380 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:35.824386 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:35.824896 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:35.825352 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:35.825355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:35.825357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:35.825487 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:35.826225 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:35.826231 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:35.826386 (+ 155us) client_negotiation.cc:769] Sending connection context
0809 20:23:35.826515 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 20:23:35.826795 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1850,"thread_start_us":1599,"threads_started":1}
I20260809 20:23:35.827760 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:35.807233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.12.78.62:38475)
0809 20:23:35.808837 (+ 1604us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:35.808843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:35.809371 (+ 528us) server_negotiation.cc:407] Connection header received
0809 20:23:35.809547 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:35.809553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:35.809635 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:35.809786 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:35.811679 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.813503 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.815075 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:35.815839 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:35.816012 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:35.820330 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:35.820338 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:35.820343 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:35.820381 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:35.824240 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:35.825025 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:35.825031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:35.825035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:35.825230 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:35.825639 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:35.825644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:35.825648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:35.826092 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:35.826247 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:35.826663 (+ 416us) server_negotiation.cc:299] Negotiation successful
0809 20:23:35.826998 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":1153,"thread_start_us":893,"threads_started":1}
I20260809 20:23:35.834540 13786 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:36586:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 20:23:35.836048 13786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:23:35.844957 13928 tablet_service.cc:1467] Processing CreateTablet for tablet 72fe0d2284cb45caaa066926ac65e467 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=521f3d17752a4dba8d299e5a6b25ec4a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 20:23:35.845232 14220 tablet_service.cc:1467] Processing CreateTablet for tablet 72fe0d2284cb45caaa066926ac65e467 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=521f3d17752a4dba8d299e5a6b25ec4a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 20:23:35.845887 14220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72fe0d2284cb45caaa066926ac65e467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:35.846009 13928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72fe0d2284cb45caaa066926ac65e467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:35.849870 14077 tablet_service.cc:1467] Processing CreateTablet for tablet 72fe0d2284cb45caaa066926ac65e467 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=521f3d17752a4dba8d299e5a6b25ec4a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 20:23:35.850535 14077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72fe0d2284cb45caaa066926ac65e467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:35.855693 14318 tablet_bootstrap.cc:492] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074: Bootstrap starting.
I20260809 20:23:35.856637 14319 tablet_bootstrap.cc:492] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226: Bootstrap starting.
I20260809 20:23:35.857997 14318 tablet_bootstrap.cc:654] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:35.858337 14317 tablet_bootstrap.cc:492] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374: Bootstrap starting.
I20260809 20:23:35.859400 14319 tablet_bootstrap.cc:654] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:35.861264 14317 tablet_bootstrap.cc:654] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:35.869788 14319 tablet_bootstrap.cc:492] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226: No bootstrap required, opened a new log
I20260809 20:23:35.870266 14319 ts_tablet_manager.cc:1397] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.009s
I20260809 20:23:35.871791 14319 raft_consensus.cc:348] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.872210 14319 raft_consensus.cc:374] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:35.872370 14319 raft_consensus.cc:729] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9186a1feba4fb7a9f15c18d03ca226, State: Initialized, Role: FOLLOWER
I20260809 20:23:35.872537 14317 tablet_bootstrap.cc:492] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374: No bootstrap required, opened a new log
I20260809 20:23:35.872803 14317 ts_tablet_manager.cc:1397] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20260809 20:23:35.872864 14319 consensus_queue.cc:260] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.873625 14325 tablet_replica.cc:439] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } }
I20260809 20:23:35.873811 14317 raft_consensus.cc:348] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.874092 14325 tablet_replica.cc:442] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:35.874131 14317 raft_consensus.cc:374] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:35.874235 14317 raft_consensus.cc:729] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c00c7560664040b2299167c7570374, State: Initialized, Role: FOLLOWER
I20260809 20:23:35.875578 14319 ts_tablet_manager.cc:1428] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260809 20:23:35.874656 14317 consensus_queue.cc:260] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.876760 14317 ts_tablet_manager.cc:1428] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 20:23:35.876485 14323 tablet_replica.cc:439] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } }
I20260809 20:23:35.877091 14323 tablet_replica.cc:442] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:35.880497 14318 tablet_bootstrap.cc:492] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074: No bootstrap required, opened a new log
I20260809 20:23:35.880896 14318 ts_tablet_manager.cc:1397] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.014s
I20260809 20:23:35.882125 14318 raft_consensus.cc:348] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.882421 14318 raft_consensus.cc:374] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:35.882524 14318 raft_consensus.cc:729] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73cb4f49244243e3a27159c6497f8074, State: Initialized, Role: FOLLOWER
I20260809 20:23:35.882812 14318 consensus_queue.cc:260] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:35.883803 14318 ts_tablet_manager.cc:1428] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260809 20:23:35.883373 14336 tablet_replica.cc:439] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } }
I20260809 20:23:35.884616 14336 tablet_replica.cc:442] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:36.205055 14323 consensus_queue.cc:1035] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:36.211968 14323 consensus_queue.cc:1035] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:36.212834 14325 raft_consensus.cc:482] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:23:36.213004 14325 raft_consensus.cc:504] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:36.215010 14325 leader_election.cc:288] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73cb4f49244243e3a27159c6497f8074 (127.12.78.1:40589), 00c00c7560664040b2299167c7570374 (127.12.78.2:32969)
I20260809 20:23:36.229947 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.216106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:52153 (local address 127.12.78.1:40589)
0809 20:23:36.216354 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:36.216363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:36.217525 (+ 1162us) server_negotiation.cc:407] Connection header received
0809 20:23:36.217884 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:36.217893 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:36.217992 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:36.218070 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:36.218921 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.220583 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.226134 (+ 5551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.228004 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.228146 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.228315 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:36.229237 (+ 922us) server_negotiation.cc:299] Negotiation successful
0809 20:23:36.229492 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 20:23:36.232311 14369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.215790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:40589 (local address 127.12.78.3:52153)
0809 20:23:36.217394 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:36.217421 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:36.217659 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:36.218337 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:36.218345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:36.218382 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:36.218729 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:36.218742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.220842 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.220850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:36.225925 (+ 5075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:36.225944 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.228587 (+ 2643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.228597 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:36.228954 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.229011 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 20:23:36.230813 (+ 1802us) client_negotiation.cc:240] Negotiation successful
0809 20:23:36.231053 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1402,"thread_start_us":14052,"threads_started":1}
I20260809 20:23:36.233476 13950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72fe0d2284cb45caaa066926ac65e467" candidate_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cb4f49244243e3a27159c6497f8074" is_pre_election: true
I20260809 20:23:36.233875 13950 raft_consensus.cc:2454] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa9186a1feba4fb7a9f15c18d03ca226 in term 0.
I20260809 20:23:36.234627 14170 leader_election.cc:302] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73cb4f49244243e3a27159c6497f8074, fa9186a1feba4fb7a9f15c18d03ca226; no voters:
I20260809 20:23:36.235229 14325 raft_consensus.cc:2781] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:23:36.235399 14325 raft_consensus.cc:482] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:23:36.235612 14325 raft_consensus.cc:3037] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:36.239992 14325 raft_consensus.cc:504] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:36.240861 14325 leader_election.cc:288] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [CANDIDATE]: Term 1 election: Requested vote from peers 73cb4f49244243e3a27159c6497f8074 (127.12.78.1:40589), 00c00c7560664040b2299167c7570374 (127.12.78.2:32969)
I20260809 20:23:36.241604 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.216808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:51489 (local address 127.12.78.2:32969)
0809 20:23:36.221111 (+ 4303us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:36.221120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:36.229820 (+ 8700us) server_negotiation.cc:407] Connection header received
0809 20:23:36.230286 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:36.230303 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:36.230396 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:36.230475 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:36.231329 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.233023 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.239215 (+ 6192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.240023 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.240180 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.240324 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:36.240913 (+ 589us) server_negotiation.cc:299] Negotiation successful
0809 20:23:36.241134 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4149,"thread_start_us":4036,"threads_started":1}
I20260809 20:23:36.241698 13954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72fe0d2284cb45caaa066926ac65e467" candidate_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cb4f49244243e3a27159c6497f8074"
I20260809 20:23:36.242153 13954 raft_consensus.cc:3037] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:36.242547 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.216309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:32969 (local address 127.12.78.3:51489)
0809 20:23:36.229709 (+ 13400us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:36.229731 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:36.230090 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:36.230717 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:36.230722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:36.230753 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:36.231105 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:36.231118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.235252 (+ 4134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.235256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:36.239015 (+ 3759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:36.239028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.240226 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.240233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:36.240653 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.240687 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 20:23:36.241825 (+ 1138us) client_negotiation.cc:240] Negotiation successful
0809 20:23:36.242007 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13273,"thread_start_us":1138,"threads_started":1}
I20260809 20:23:36.243875 14103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72fe0d2284cb45caaa066926ac65e467" candidate_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c00c7560664040b2299167c7570374" is_pre_election: true
I20260809 20:23:36.244181 14103 raft_consensus.cc:2454] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa9186a1feba4fb7a9f15c18d03ca226 in term 0.
I20260809 20:23:36.243868 14102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72fe0d2284cb45caaa066926ac65e467" candidate_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c00c7560664040b2299167c7570374"
I20260809 20:23:36.244910 14102 raft_consensus.cc:3037] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:36.247025 13954 raft_consensus.cc:2454] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa9186a1feba4fb7a9f15c18d03ca226 in term 1.
I20260809 20:23:36.247524 14170 leader_election.cc:302] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73cb4f49244243e3a27159c6497f8074, fa9186a1feba4fb7a9f15c18d03ca226; no voters:
I20260809 20:23:36.248080 14325 raft_consensus.cc:2781] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:36.248670 14102 raft_consensus.cc:2454] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa9186a1feba4fb7a9f15c18d03ca226 in term 1.
I20260809 20:23:36.257067 14325 raft_consensus.cc:686] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [term 1 LEADER]: Becoming Leader. State: Replica: fa9186a1feba4fb7a9f15c18d03ca226, State: Running, Role: LEADER
I20260809 20:23:36.257725 14325 consensus_queue.cc:237] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } }
I20260809 20:23:36.261842 14372 tablet_replica.cc:439] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226: TxnStatusTablet state changed. Reason: New leader fa9186a1feba4fb7a9f15c18d03ca226. Latest consensus state: current_term: 1 leader_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } }
I20260809 20:23:36.262452 14372 tablet_replica.cc:442] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226: This TxnStatusTablet replica's current role is: LEADER
I20260809 20:23:36.263662 14374 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 20:23:36.264030 14374 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 20:23:36.264304 13786 catalog_manager.cc:5495] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa9186a1feba4fb7a9f15c18d03ca226 (127.12.78.3). New cstate: current_term: 1 leader_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } health_report { overall_health: HEALTHY } } }
I20260809 20:23:36.330746 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.322560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:34137 (local address 127.0.0.1:34668)
0809 20:23:36.322910 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:36.322935 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:36.323158 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:36.324130 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:36.324135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:36.324154 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 20:23:36.324452 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:36.324461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.326120 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.326123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:36.327426 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:36.327451 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.328867 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.328871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:36.328991 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.329052 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 20:23:36.330315 (+ 1263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 20:23:36.330332 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 20:23:36.330386 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 20:23:36.330509 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 20:23:36.330914 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.322641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.12.78.3:34137)
0809 20:23:36.323475 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:36.323482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:36.323507 (+ 25us) server_negotiation.cc:407] Connection header received
0809 20:23:36.323618 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:36.323624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:36.323736 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:36.324045 (+ 309us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:23:36.324667 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.325979 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.327597 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.328648 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.328844 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.329298 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:23:36.330199 (+ 901us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:23:36.330362 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:36.330435 (+ 73us) server_negotiation.cc:299] Negotiation successful
0809 20:23:36.330520 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":491,"threads_started":1}
I20260809 20:23:36.335513 14102 raft_consensus.cc:1261] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374 [term 1 FOLLOWER]: Refusing update from remote peer fa9186a1feba4fb7a9f15c18d03ca226: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:36.335585 13954 raft_consensus.cc:1261] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074 [term 1 FOLLOWER]: Refusing update from remote peer fa9186a1feba4fb7a9f15c18d03ca226: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:36.336355 14372 consensus_queue.cc:1035] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:36.336849 14325 consensus_queue.cc:1035] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:36.341094 14323 tablet_replica.cc:439] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374: TxnStatusTablet state changed. Reason: New leader fa9186a1feba4fb7a9f15c18d03ca226. Latest consensus state: current_term: 1 leader_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } }
I20260809 20:23:36.341560 14323 tablet_replica.cc:442] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:36.343940 14366 mvcc.cc:205] Tried to move back new op lower bound from 7316713538902835200 to 7316713538596343808. Current Snapshot: MvccSnapshot[applied={T|T < 7316713538902835200}]
I20260809 20:23:36.345604 14323 tablet_replica.cc:439] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } }
I20260809 20:23:36.346065 14323 tablet_replica.cc:442] T 72fe0d2284cb45caaa066926ac65e467 P 00c00c7560664040b2299167c7570374: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:36.353147 14336 tablet_replica.cc:439] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074: TxnStatusTablet state changed. Reason: New leader fa9186a1feba4fb7a9f15c18d03ca226. Latest consensus state: current_term: 1 leader_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } }
I20260809 20:23:36.353585 14336 tablet_replica.cc:442] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:36.353843 14372 tablet_replica.cc:439] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } }
I20260809 20:23:36.355439 14372 tablet_replica.cc:442] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226: This TxnStatusTablet replica's current role is: LEADER
I20260809 20:23:36.358834 14325 tablet_replica.cc:439] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } }
I20260809 20:23:36.360746 14325 tablet_replica.cc:442] T 72fe0d2284cb45caaa066926ac65e467 P fa9186a1feba4fb7a9f15c18d03ca226: This TxnStatusTablet replica's current role is: LEADER
I20260809 20:23:36.369572 14336 tablet_replica.cc:439] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cb4f49244243e3a27159c6497f8074" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40589 } } peers { permanent_uuid: "00c00c7560664040b2299167c7570374" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 32969 } } peers { permanent_uuid: "fa9186a1feba4fb7a9f15c18d03ca226" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34137 } } }
I20260809 20:23:36.369876 14336 tablet_replica.cc:442] T 72fe0d2284cb45caaa066926ac65e467 P 73cb4f49244243e3a27159c6497f8074: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:36.373382 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.364642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.12.78.2:32969)
0809 20:23:36.364940 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:36.364949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:36.365638 (+ 689us) server_negotiation.cc:407] Connection header received
0809 20:23:36.365879 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:36.365886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:36.366082 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:36.366271 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:23:36.366843 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.367944 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.370579 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.371426 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.371537 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.372178 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:23:36.372795 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:23:36.372949 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:36.373031 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 20:23:36.373119 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 20:23:36.391360 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.384881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:51201 (local address 127.12.78.3:34137)
0809 20:23:36.385114 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:36.385121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:36.385381 (+ 260us) server_negotiation.cc:407] Connection header received
0809 20:23:36.385544 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:36.385550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:36.385640 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:36.385742 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:23:36.386317 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.387559 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.389273 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.389982 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.390110 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.390573 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:23:36.390710 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:23:36.390817 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:36.390918 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 20:23:36.391039 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 20:23:36.391510 14388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.384206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:34137 (local address 127.12.78.2:51201)
0809 20:23:36.385143 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:36.385164 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:36.385409 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:36.385761 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:36.385764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:36.385773 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 20:23:36.386181 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:36.386190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.387742 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.387745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:36.389126 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:36.389138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.390149 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.390153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:36.390307 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.390385 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 20:23:36.390819 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 20:23:36.390827 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 20:23:36.390888 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 20:23:36.391030 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":748,"threads_started":1}
I20260809 20:23:36.439931 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.431130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:50429 (local address 127.12.78.2:32969)
0809 20:23:36.431459 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:36.431468 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:36.432771 (+ 1303us) server_negotiation.cc:407] Connection header received
0809 20:23:36.433047 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:36.433054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:36.433135 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:36.433234 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:23:36.434091 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.435181 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.437175 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.437927 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.438037 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.438796 (+ 759us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:23:36.438962 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:23:36.439033 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:36.439259 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 20:23:36.439343 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260809 20:23:36.440047 14391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.431030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:32969 (local address 127.12.78.3:50429)
0809 20:23:36.432463 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:36.432485 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:36.432910 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:36.433265 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:36.433269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:36.433283 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 20:23:36.433883 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:36.433898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.435319 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.435326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:36.437000 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:36.437015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.438221 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.438227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:36.438476 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.438565 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 20:23:36.439124 (+ 559us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 20:23:36.439132 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 20:23:36.439230 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 20:23:36.439396 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"thread_start_us":1567,"threads_started":1}
I20260809 20:23:36.486020 13787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:23:36.486394 13787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:23:36.490531 13787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a518b8ffed0449f9906c87100f93a5c: Sending DeleteTablet for 3 replicas of tablet 46eab50d90fd41c291582f78077ab53d
I20260809 20:23:36.491602 13928 tablet_service.cc:1514] Processing DeleteTablet for tablet 46eab50d90fd41c291582f78077ab53d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:36 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:59962
I20260809 20:23:36.491662 14077 tablet_service.cc:1514] Processing DeleteTablet for tablet 46eab50d90fd41c291582f78077ab53d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:36 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:43864
I20260809 20:23:36.493168 14220 tablet_service.cc:1514] Processing DeleteTablet for tablet 46eab50d90fd41c291582f78077ab53d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:36 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:34664
I20260809 20:23:36.499369 14397 tablet_replica.cc:330] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374: stopping tablet replica
I20260809 20:23:36.504608 14396 tablet_replica.cc:330] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074: stopping tablet replica
I20260809 20:23:36.504995 14396 raft_consensus.cc:2229] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:23:36.505043 14397 raft_consensus.cc:2229] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:23:36.505367 14396 raft_consensus.cc:2258] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:36.506280 14397 raft_consensus.cc:2258] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:36.506875 14398 tablet_replica.cc:330] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226: stopping tablet replica
I20260809 20:23:36.507220 14398 raft_consensus.cc:2229] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:23:36.507879 14398 raft_consensus.cc:2258] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:36.508219 14397 ts_tablet_manager.cc:1905] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:36.509292 14396 ts_tablet_manager.cc:1905] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:36.513659 14398 ts_tablet_manager.cc:1905] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:36.516106 14396 ts_tablet_manager.cc:1918] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:23:36.516275 14396 log.cc:1196] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/46eab50d90fd41c291582f78077ab53d
I20260809 20:23:36.516880 14396 ts_tablet_manager.cc:1939] T 46eab50d90fd41c291582f78077ab53d P 73cb4f49244243e3a27159c6497f8074: Deleting consensus metadata
I20260809 20:23:36.524245 14398 ts_tablet_manager.cc:1918] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:23:36.524612 14398 log.cc:1196] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/46eab50d90fd41c291582f78077ab53d
I20260809 20:23:36.525173 13771 catalog_manager.cc:4841] TS 73cb4f49244243e3a27159c6497f8074 (127.12.78.1:40589): tablet 46eab50d90fd41c291582f78077ab53d (table test-table [id=b70e38ad30ec4e28bf23b162988ff828]) successfully deleted
I20260809 20:23:36.525343 14398 ts_tablet_manager.cc:1939] T 46eab50d90fd41c291582f78077ab53d P fa9186a1feba4fb7a9f15c18d03ca226: Deleting consensus metadata
I20260809 20:23:36.526892 13772 catalog_manager.cc:4841] TS fa9186a1feba4fb7a9f15c18d03ca226 (127.12.78.3:34137): tablet 46eab50d90fd41c291582f78077ab53d (table test-table [id=b70e38ad30ec4e28bf23b162988ff828]) successfully deleted
I20260809 20:23:36.533087 14397 ts_tablet_manager.cc:1918] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:23:36.533284 14397 log.cc:1196] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/46eab50d90fd41c291582f78077ab53d
I20260809 20:23:36.533941 14397 ts_tablet_manager.cc:1939] T 46eab50d90fd41c291582f78077ab53d P 00c00c7560664040b2299167c7570374: Deleting consensus metadata
I20260809 20:23:36.535470 13773 catalog_manager.cc:4841] TS 00c00c7560664040b2299167c7570374 (127.12.78.2:32969): tablet 46eab50d90fd41c291582f78077ab53d (table test-table [id=b70e38ad30ec4e28bf23b162988ff828]) successfully deleted
Aug 09 20:23:36 dist-test-slave-rnrw krb5kdc[13741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:23:36.569186 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.524835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.12.78.1:40589)
0809 20:23:36.525610 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:36.525617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:36.541704 (+ 16087us) server_negotiation.cc:407] Connection header received
0809 20:23:36.542355 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:36.542362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:36.542461 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:36.542629 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:36.545672 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.547120 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.549872 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.550930 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.551061 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.558732 (+ 7671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:36.558793 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:36.558815 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:36.558880 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:36.564655 (+ 5775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:36.566023 (+ 1368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:36.566033 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:36.566037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:36.566235 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:36.566854 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:36.566859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:36.566863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:36.567371 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:36.567973 (+ 602us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:36.568243 (+ 270us) server_negotiation.cc:299] Negotiation successful
0809 20:23:36.568649 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573}
W20260809 20:23:36.570348 14000 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59986
I20260809 20:23:36.620321 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.578429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.12.78.62:38475)
0809 20:23:36.594044 (+ 15615us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:36.594054 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:36.597090 (+ 3036us) server_negotiation.cc:407] Connection header received
0809 20:23:36.597363 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:36.597370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:36.597503 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:36.597623 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:36.600462 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.602139 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.604452 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.605250 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.605385 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.610487 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:36.610496 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:36.610501 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:36.610570 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:36.615335 (+ 4765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:36.616471 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:36.616478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:36.616482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:36.616741 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:36.617270 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:36.617276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:36.617281 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:36.617814 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:36.618083 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:36.619485 (+ 1402us) server_negotiation.cc:299] Negotiation successful
0809 20:23:36.619783 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15379,"thread_start_us":15645,"threads_started":1}
W20260809 20:23:36.621003 13787 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36592
I20260809 20:23:36.623777 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 13858
I20260809 20:23:36.645200 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.624897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59996 (local address 127.12.78.1:40589)
0809 20:23:36.625221 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:36.625227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:36.627132 (+ 1905us) server_negotiation.cc:407] Connection header received
0809 20:23:36.627298 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:36.627304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:36.627388 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:36.627514 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:36.629285 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.630632 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.632717 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.633360 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.633485 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.637114 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:36.637121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:36.637126 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:36.637168 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:36.641475 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:36.642359 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:36.642367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:36.642370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:36.642607 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:36.643172 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:36.643177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:36.643181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:36.643683 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:36.644059 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:36.644565 (+ 506us) server_negotiation.cc:299] Negotiation successful
0809 20:23:36.644838 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
W20260809 20:23:36.646298 14000 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59996
W20260809 20:23:36.647105 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 13858: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:23:36.647312 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 13858
I20260809 20:23:36.683616 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 14009
Aug 09 20:23:36 dist-test-slave-rnrw krb5kdc[13741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:23:36.708179 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.685015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.12.78.2:32969)
0809 20:23:36.685707 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:36.685714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:36.685735 (+ 21us) server_negotiation.cc:407] Connection header received
0809 20:23:36.685803 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:36.685808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:36.685912 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:36.686030 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:36.688101 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.689468 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.691566 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.692365 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.692527 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.698861 (+ 6334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:36.698880 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:36.698903 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:36.698969 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:36.703763 (+ 4794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:36.704918 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:36.704928 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:36.704932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:36.705097 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:36.705644 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:36.705650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:36.705653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:36.706353 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:36.706787 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:36.707494 (+ 707us) server_negotiation.cc:299] Negotiation successful
0809 20:23:36.707780 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542}
W20260809 20:23:36.710170 14144 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43878
W20260809 20:23:36.711134 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 14009: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:23:36.711325 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 14009
I20260809 20:23:36.733938 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 14153
Aug 09 20:23:36 dist-test-slave-rnrw krb5kdc[13741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:23:36.751724 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:36.734844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34674 (local address 127.12.78.3:34137)
0809 20:23:36.735065 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:36.735072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:36.735094 (+ 22us) server_negotiation.cc:407] Connection header received
0809 20:23:36.735212 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:36.735217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:36.735307 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:36.735433 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:36.736712 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.737869 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.739338 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:36.740010 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:36.740146 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:36.744363 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:36.744377 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:36.744396 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:36.744439 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:36.748789 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:36.749570 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:36.749581 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:36.749585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:36.749738 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:36.750093 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:36.750098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:36.750101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:36.750556 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:36.750911 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:36.751123 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 20:23:36.751360 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260809 20:23:36.752591 14291 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34674
W20260809 20:23:36.753372 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 14153: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:23:36.753553 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 14153
I20260809 20:23:36.772437 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 13754
I20260809 20:23:36.773591 13827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:36.891904 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 13754
2026-08-09T20:23:36Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5188 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:23:36 dist-test-slave-rnrw krb5kdc[14408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:23:36 dist-test-slave-rnrw krb5kdc[14408](info): set up 2 sockets
Aug 09 20:23:36 dist-test-slave-rnrw krb5kdc[14408](info): commencing operation
krb5kdc: starting...
W20260809 20:23:38.024195 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:23:38 dist-test-slave-rnrw krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:23:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:23:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:23:38.236263 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:41195
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:41195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:38.575973 14421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:38.576270 14421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:38.576366 14421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:38.583997 14421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:23:38.584102 14421 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 20:23:38.584143 14421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:38.584204 14421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:38.584254 14421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:23:38.584307 14421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:23:38.594151 14421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:41195
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:41195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:38.595479 14421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:38.596755 14421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:38.608233 14426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:38.608249 14429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:38.608253 14427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:38.609124 14421 server_base.cc:1034] running on GCE node
I20260809 20:23:38.609839 14421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:38.611158 14421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:38.612372 14421 hybrid_clock.cc:648] HybridClock initialized: now 1786307018612315 us; error 31 us; skew 500 ppm
I20260809 20:23:38.612720 14421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:38 dist-test-slave-rnrw krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307018, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:38.621060 14421 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:23:38.622613 14421 webserver.cc:460] Webserver started at http://127.12.78.62:33635/ using document root <none> and password file <none>
I20260809 20:23:38.623358 14421 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:38.623469 14421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:38.623804 14421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:38.626924 14421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "950311ec2bfe4fc983d4596f79fcfe39"
format_stamp: "Formatted at 2026-08-09 20:23:38 on dist-test-slave-rnrw"
server_key: "870f354532155375e78992682c27f0a8"
server_key_iv: "83f6a4d7be77d01fc11ebd5b2b0f3200"
server_key_version: "encryptionkey@0"
I20260809 20:23:38.627712 14421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "950311ec2bfe4fc983d4596f79fcfe39"
format_stamp: "Formatted at 2026-08-09 20:23:38 on dist-test-slave-rnrw"
server_key: "870f354532155375e78992682c27f0a8"
server_key_iv: "83f6a4d7be77d01fc11ebd5b2b0f3200"
server_key_version: "encryptionkey@0"
I20260809 20:23:38.632895 14421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260809 20:23:38.636211 14436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:38.637511 14421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:23:38.637712 14421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "950311ec2bfe4fc983d4596f79fcfe39"
format_stamp: "Formatted at 2026-08-09 20:23:38 on dist-test-slave-rnrw"
server_key: "870f354532155375e78992682c27f0a8"
server_key_iv: "83f6a4d7be77d01fc11ebd5b2b0f3200"
server_key_version: "encryptionkey@0"
I20260809 20:23:38.637924 14421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:38.655541 14421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:38.659528 14421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:38.659816 14421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:38.681375 14421 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:41195
I20260809 20:23:38.681382 14500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:41195 every 8 connection(s)
I20260809 20:23:38.682757 14421 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:23:38.686508 14501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:38.687314 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 14421
I20260809 20:23:38.687551 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:23:38.688428 12600 external_mini_cluster.cc:1428] Setting key ad251f6f183f795fcda3b842060dda82
I20260809 20:23:38.700960 14501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39: Bootstrap starting.
Aug 09 20:23:38 dist-test-slave-rnrw krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:38.704722 14501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:38.705989 14501 log.cc:824] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:38.708917 14501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39: No bootstrap required, opened a new log
I20260809 20:23:38.711485 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:38.690546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41804 (local address 127.12.78.62:41195)
0809 20:23:38.691373 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:38.691384 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:38.691428 (+ 44us) server_negotiation.cc:407] Connection header received
0809 20:23:38.693090 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:38.693110 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:38.693495 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:38.693831 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:38.695633 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:38.696963 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:38.698822 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:38.699472 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:38.699597 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:38.704301 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:38.704309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:38.704323 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:38.704357 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:38.707522 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:38.708194 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:38.708200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:38.708205 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:38.708374 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:38.708821 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:38.708824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:38.708826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:38.709386 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:38.709690 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:38.710111 (+ 421us) server_negotiation.cc:299] Negotiation successful
0809 20:23:38.710537 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":387,"threads_started":1}
I20260809 20:23:38.716739 14501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950311ec2bfe4fc983d4596f79fcfe39" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41195 } }
I20260809 20:23:38.717079 14501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:38.717199 14501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950311ec2bfe4fc983d4596f79fcfe39, State: Initialized, Role: FOLLOWER
I20260809 20:23:38.717737 14501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950311ec2bfe4fc983d4596f79fcfe39" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41195 } }
I20260809 20:23:38.717967 14501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:23:38.718118 14501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:23:38.718286 14501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:38.720726 14501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950311ec2bfe4fc983d4596f79fcfe39" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41195 } }
I20260809 20:23:38.721244 14501 leader_election.cc:302] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 950311ec2bfe4fc983d4596f79fcfe39; no voters:
I20260809 20:23:38.721817 14501 leader_election.cc:288] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:23:38.721974 14506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:38.722695 14506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [term 1 LEADER]: Becoming Leader. State: Replica: 950311ec2bfe4fc983d4596f79fcfe39, State: Running, Role: LEADER
I20260809 20:23:38.723191 14506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950311ec2bfe4fc983d4596f79fcfe39" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41195 } }
I20260809 20:23:38.723919 14501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:23:38.726068 14508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 950311ec2bfe4fc983d4596f79fcfe39. Latest consensus state: current_term: 1 leader_uuid: "950311ec2bfe4fc983d4596f79fcfe39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950311ec2bfe4fc983d4596f79fcfe39" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41195 } } }
I20260809 20:23:38.726459 14508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:38.726030 14507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "950311ec2bfe4fc983d4596f79fcfe39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950311ec2bfe4fc983d4596f79fcfe39" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41195 } } }
I20260809 20:23:38.727272 14507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39 [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:38.729956 14513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:23:38.735136 14513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:23:38.744282 14513 catalog_manager.cc:1347] Generated new cluster ID: cd0740cdab214c6f80dc505bc9027881
I20260809 20:23:38.744390 14513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:23:38.766969 14513 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:23:38.768910 14513 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:23:38.781831 14513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39: Generated new TSK 0
I20260809 20:23:38.783098 14513 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:23:38.838297 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:41195
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:39.184630 14527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:39.184918 14527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:39.185104 14527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:39.192526 14527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:39.192691 14527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:39.192865 14527 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 20:23:39.192950 14527 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 20:23:39.193084 14527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:23:39.203162 14527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.78.62:41195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:39.204679 14527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:39.205997 14527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:39.219748 14532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:39.219846 14537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:39.219892 14535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:39.220788 14527 server_base.cc:1034] running on GCE node
I20260809 20:23:39.221377 14527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:39.222362 14527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:39.223572 14527 hybrid_clock.cc:648] HybridClock initialized: now 1786307019223516 us; error 46 us; skew 500 ppm
I20260809 20:23:39.223968 14527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:39 dist-test-slave-rnrw krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307019, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:39.227524 14527 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:23:39.229169 14527 webserver.cc:460] Webserver started at http://127.12.78.1:43207/ using document root <none> and password file <none>
I20260809 20:23:39.230000 14527 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:39.230170 14527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:39.230548 14527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:39.233601 14527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "ff8264ce544740fbaafde6a9e59659d1"
format_stamp: "Formatted at 2026-08-09 20:23:39 on dist-test-slave-rnrw"
server_key: "b34adcbaa612a2dbb5705ee86c68d6a9"
server_key_iv: "7a0a1de6ded23bc8a62fc96eb4107942"
server_key_version: "encryptionkey@0"
I20260809 20:23:39.234740 14527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "ff8264ce544740fbaafde6a9e59659d1"
format_stamp: "Formatted at 2026-08-09 20:23:39 on dist-test-slave-rnrw"
server_key: "b34adcbaa612a2dbb5705ee86c68d6a9"
server_key_iv: "7a0a1de6ded23bc8a62fc96eb4107942"
server_key_version: "encryptionkey@0"
I20260809 20:23:39.240185 14527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:23:39.243778 14544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:39.245162 14527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 20:23:39.245347 14527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "ff8264ce544740fbaafde6a9e59659d1"
format_stamp: "Formatted at 2026-08-09 20:23:39 on dist-test-slave-rnrw"
server_key: "b34adcbaa612a2dbb5705ee86c68d6a9"
server_key_iv: "7a0a1de6ded23bc8a62fc96eb4107942"
server_key_version: "encryptionkey@0"
I20260809 20:23:39.245558 14527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:39.263208 14527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:39.267997 14527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:39.268338 14527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:39.271183 14527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:39.271347 14527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:39.271497 14527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:39.271620 14527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 20:23:39 dist-test-slave-rnrw krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307019, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:39.297590 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:39.271758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:41339 (local address 127.12.78.62:41195)
0809 20:23:39.272509 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:39.272516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:39.273613 (+ 1097us) server_negotiation.cc:407] Connection header received
0809 20:23:39.276740 (+ 3127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:39.276749 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:39.276841 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:39.277003 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:39.279390 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.280784 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.283328 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.284118 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.284285 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:39.289844 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:39.289850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:39.289853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:39.289889 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:39.293254 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:39.294482 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:39.294491 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:39.294494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:39.294669 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:39.295277 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:39.295286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:39.295292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:39.295819 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:39.296012 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:39.296787 (+ 775us) server_negotiation.cc:299] Negotiation successful
0809 20:23:39.297093 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":507,"threads_started":1}
I20260809 20:23:39.299230 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:39.272383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:41195 (local address 127.12.78.1:41339)
0809 20:23:39.273409 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:39.273474 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:39.276331 (+ 2857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:39.277353 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:39.277368 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:39.278254 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:39.279150 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:39.279187 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.281024 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.281030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:39.283169 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:39.283182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.284665 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.284672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:39.285137 (+ 465us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:39.286110 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:39.286160 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:39.289560 (+ 3400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:39.293422 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:39.293438 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:39.293461 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:39.294318 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:39.294802 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:39.294808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:39.294811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:39.295102 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:39.295983 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:39.295989 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:39.296345 (+ 356us) client_negotiation.cc:769] Sending connection context
0809 20:23:39.296673 (+ 328us) client_negotiation.cc:240] Negotiation successful
0809 20:23:39.297405 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":430,"threads_started":1}
I20260809 20:23:39.321890 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:39.305178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:41195 (local address 127.12.78.1:44307)
0809 20:23:39.305940 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:39.305965 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:39.306141 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:39.306478 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:39.306486 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:39.307250 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:39.307778 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:39.307795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.308851 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.308857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:39.310696 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:39.310712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.311687 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.311697 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:39.311962 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:39.312899 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:39.312933 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:39.314381 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:39.318691 (+ 4310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:39.318705 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:39.318712 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:39.319530 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:39.320024 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:39.320028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:39.320032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:39.320222 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:39.320996 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:39.321004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:39.321200 (+ 196us) client_negotiation.cc:769] Sending connection context
0809 20:23:39.321317 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 20:23:39.321555 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":386,"threads_started":1}
I20260809 20:23:39.322309 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:39.305214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:44307 (local address 127.12.78.62:41195)
0809 20:23:39.305442 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:39.305451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:39.306065 (+ 614us) server_negotiation.cc:407] Connection header received
0809 20:23:39.306262 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:39.306265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:39.306329 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:39.306409 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:39.307937 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.308718 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.310851 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.311435 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.311584 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:39.314632 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:39.314637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:39.314641 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:39.314677 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:39.318488 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:39.319686 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:39.319693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:39.319699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:39.319888 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:39.320355 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:39.320360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:39.320364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:39.320838 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:39.321034 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:39.321492 (+ 458us) server_negotiation.cc:299] Negotiation successful
0809 20:23:39.321821 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 20:23:39.331873 14527 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:35113
I20260809 20:23:39.331902 14672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:35113 every 8 connection(s)
I20260809 20:23:39.333206 14527 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:23:39.335265 14673 heartbeater.cc:347] Connected to a master server at 127.12.78.62:41195
I20260809 20:23:39.335501 14673 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:39.336139 14673 heartbeater.cc:511] Master 127.12.78.62:41195 requested a full tablet report, sending...
I20260809 20:23:39.338136 14455 ts_manager.cc:194] Registered new tserver with Master: ff8264ce544740fbaafde6a9e59659d1 (127.12.78.1:35113)
I20260809 20:23:39.339581 14455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:41339
I20260809 20:23:39.342996 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 14527
I20260809 20:23:39.343216 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:23:39.343856 12600 external_mini_cluster.cc:1428] Setting key 9960f6908c3888f19f5a74c24642fc83
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:23:39.389367 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:41195
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:39.729720 14677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:39.730044 14677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:39.730152 14677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:39.737638 14677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:39.737805 14677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:39.737905 14677 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 20:23:39.737959 14677 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 20:23:39.738031 14677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:23:39.747772 14677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.78.62:41195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:39.749146 14677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:39.750660 14677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:39.763691 14682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:39.763720 14684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:39.763824 14686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:39.764667 14677 server_base.cc:1034] running on GCE node
I20260809 20:23:39.765209 14677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:39.766080 14677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:39.767275 14677 hybrid_clock.cc:648] HybridClock initialized: now 1786307019767222 us; error 53 us; skew 500 ppm
I20260809 20:23:39.767596 14677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:39 dist-test-slave-rnrw krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307019, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:39.770704 14677 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:23:39.772119 14677 webserver.cc:460] Webserver started at http://127.12.78.2:38689/ using document root <none> and password file <none>
I20260809 20:23:39.772873 14677 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:39.773023 14677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:39.773352 14677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:39.776285 14677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "2966bfabe5cf4693bf89c970c3426e69"
format_stamp: "Formatted at 2026-08-09 20:23:39 on dist-test-slave-rnrw"
server_key: "fa6d671bb06058d2de93b094d1990e9e"
server_key_iv: "b992bf4cbb2f5a1c5cae5541df171810"
server_key_version: "encryptionkey@0"
I20260809 20:23:39.777082 14677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "2966bfabe5cf4693bf89c970c3426e69"
format_stamp: "Formatted at 2026-08-09 20:23:39 on dist-test-slave-rnrw"
server_key: "fa6d671bb06058d2de93b094d1990e9e"
server_key_iv: "b992bf4cbb2f5a1c5cae5541df171810"
server_key_version: "encryptionkey@0"
I20260809 20:23:39.782061 14677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 20:23:39.785392 14693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:39.786639 14677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:23:39.786826 14677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "2966bfabe5cf4693bf89c970c3426e69"
format_stamp: "Formatted at 2026-08-09 20:23:39 on dist-test-slave-rnrw"
server_key: "fa6d671bb06058d2de93b094d1990e9e"
server_key_iv: "b992bf4cbb2f5a1c5cae5541df171810"
server_key_version: "encryptionkey@0"
I20260809 20:23:39.787024 14677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:39.809306 14677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:39.813270 14677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:39.813566 14677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:39.816443 14677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:39.816598 14677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:39.816772 14677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:39.816875 14677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 20:23:39 dist-test-slave-rnrw krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307019, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:39.839484 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:39.816836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:37379 (local address 127.12.78.62:41195)
0809 20:23:39.817044 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:39.817052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:39.818594 (+ 1542us) server_negotiation.cc:407] Connection header received
0809 20:23:39.821654 (+ 3060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:39.821661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:39.821748 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:39.821861 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:39.824179 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.825357 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.827897 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.828355 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.828493 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:39.832242 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:39.832251 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:39.832259 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:39.832291 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:39.835499 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:39.836247 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:39.836252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:39.836256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:39.836384 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:39.836897 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:39.836904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:39.836910 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:39.837343 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:39.837514 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:39.838585 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0809 20:23:39.838940 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 20:23:39.840137 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:39.817466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:41195 (local address 127.12.78.2:37379)
0809 20:23:39.818412 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:39.818462 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:39.821302 (+ 2840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:39.822192 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:39.822204 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:39.822925 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:39.823970 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:39.824001 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.825506 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.825513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:39.827742 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:39.827760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.828665 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.828676 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:39.828881 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:39.829581 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:39.829607 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:39.832026 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:39.835638 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:39.835651 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:39.835674 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:39.836107 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:39.836498 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:39.836503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:39.836507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:39.836768 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:39.837490 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:39.837498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:39.838040 (+ 542us) client_negotiation.cc:769] Sending connection context
0809 20:23:39.838432 (+ 392us) client_negotiation.cc:240] Negotiation successful
0809 20:23:39.838957 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":383,"threads_started":1}
I20260809 20:23:39.863197 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:39.846683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:33989 (local address 127.12.78.62:41195)
0809 20:23:39.846955 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:39.846964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:39.847649 (+ 685us) server_negotiation.cc:407] Connection header received
0809 20:23:39.847865 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:39.847870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:39.847950 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:39.848030 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:39.849525 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.850637 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.853309 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.853805 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.853955 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:39.856899 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:39.856906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:39.856911 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:39.856951 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:39.859924 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:39.860962 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:39.860966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:39.860968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:39.861100 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:39.861577 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:39.861583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:39.861588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:39.862017 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:39.862191 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:39.862615 (+ 424us) server_negotiation.cc:299] Negotiation successful
0809 20:23:39.862853 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 20:23:39.863312 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:39.846667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:41195 (local address 127.12.78.2:33989)
0809 20:23:39.847508 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:39.847541 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:39.847720 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:39.848097 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:39.848105 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:39.848873 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:39.849356 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:39.849374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.850848 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.850864 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:39.853101 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:39.853123 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:39.854006 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:39.854012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:39.854282 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:39.855072 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:39.855113 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:39.856685 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:39.860069 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:39.860078 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:39.860085 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:39.860817 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:39.861234 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:39.861241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:39.861247 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:39.861447 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:39.862170 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:39.862176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:39.862362 (+ 186us) client_negotiation.cc:769] Sending connection context
0809 20:23:39.862503 (+ 141us) client_negotiation.cc:240] Negotiation successful
0809 20:23:39.862854 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":439,"threads_started":1}
I20260809 20:23:39.878116 14677 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:42703
I20260809 20:23:39.878158 14814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:42703 every 8 connection(s)
I20260809 20:23:39.879418 14677 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:23:39.879766 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 14677
I20260809 20:23:39.879952 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:23:39.880533 12600 external_mini_cluster.cc:1428] Setting key d0474d319a4a72f8f4b99abefbb324b4
I20260809 20:23:39.882015 14815 heartbeater.cc:347] Connected to a master server at 127.12.78.62:41195
I20260809 20:23:39.882210 14815 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:39.882897 14815 heartbeater.cc:511] Master 127.12.78.62:41195 requested a full tablet report, sending...
I20260809 20:23:39.884358 14454 ts_manager.cc:194] Registered new tserver with Master: 2966bfabe5cf4693bf89c970c3426e69 (127.12.78.2:42703)
I20260809 20:23:39.885278 14454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:37379
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:23:39.928512 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:41195
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:40.265442 14819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:40.265825 14819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:40.266008 14819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:40.273538 14819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:40.273739 14819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:40.273906 14819 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 20:23:40.273998 14819 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 20:23:40.274135 14819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:23:40.284283 14819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.78.62:41195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:40.285797 14819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:40.287691 14819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:40.302493 14826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:40.302564 14827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:40.302536 14829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:40.303419 14819 server_base.cc:1034] running on GCE node
I20260809 20:23:40.304119 14819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:40.305192 14819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:40.306419 14819 hybrid_clock.cc:648] HybridClock initialized: now 1786307020306372 us; error 28 us; skew 500 ppm
I20260809 20:23:40.306818 14819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:40 dist-test-slave-rnrw krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307020, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:40.310420 14819 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:23:40.312130 14819 webserver.cc:460] Webserver started at http://127.12.78.3:40465/ using document root <none> and password file <none>
I20260809 20:23:40.312996 14819 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:40.313166 14819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:40.313529 14819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:40.316881 14819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "ebc20402607e47b39946bf6014a57c95"
format_stamp: "Formatted at 2026-08-09 20:23:40 on dist-test-slave-rnrw"
server_key: "b080ea8cd2baf3d3001a92117dfd45b4"
server_key_iv: "9e01cb600e5c557b8a445990ff4f0fa1"
server_key_version: "encryptionkey@0"
I20260809 20:23:40.317752 14819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "ebc20402607e47b39946bf6014a57c95"
format_stamp: "Formatted at 2026-08-09 20:23:40 on dist-test-slave-rnrw"
server_key: "b080ea8cd2baf3d3001a92117dfd45b4"
server_key_iv: "9e01cb600e5c557b8a445990ff4f0fa1"
server_key_version: "encryptionkey@0"
I20260809 20:23:40.323400 14819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:23:40.327447 14837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:40.328958 14819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:23:40.329165 14819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "ebc20402607e47b39946bf6014a57c95"
format_stamp: "Formatted at 2026-08-09 20:23:40 on dist-test-slave-rnrw"
server_key: "b080ea8cd2baf3d3001a92117dfd45b4"
server_key_iv: "9e01cb600e5c557b8a445990ff4f0fa1"
server_key_version: "encryptionkey@0"
I20260809 20:23:40.329427 14819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:40.342377 14673 heartbeater.cc:503] Master 127.12.78.62:41195 was elected leader, sending a full tablet report...
I20260809 20:23:40.344753 14819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:40.349283 14819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:40.349586 14819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:40.352653 14819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:40.352802 14819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:40.352921 14819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:40.353024 14819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 20:23:40 dist-test-slave-rnrw krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307020, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:40.375751 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.353006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:46585 (local address 127.12.78.62:41195)
0809 20:23:40.353208 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.353217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.354642 (+ 1425us) server_negotiation.cc:407] Connection header received
0809 20:23:40.356730 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.356736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.356835 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.356948 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:40.359099 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.359860 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.362033 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.362624 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.362766 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.367930 (+ 5164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:40.367938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:40.367942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:40.367983 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:40.371826 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:40.372599 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:40.372607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:40.372613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:40.372800 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:40.373257 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:40.373262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:40.373267 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:40.373657 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:40.373900 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.374887 (+ 987us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.375243 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 20:23:40.376452 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.353569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:41195 (local address 127.12.78.3:46585)
0809 20:23:40.354465 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:40.354509 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:40.356365 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:40.357199 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:40.357207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:40.358043 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:40.358927 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.358946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.360020 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.360031 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.361864 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.361890 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.362904 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.362914 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.363247 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.364260 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:40.364297 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:40.367656 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:40.371969 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:40.371978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:40.372000 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:40.372443 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:40.372919 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:40.372924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:40.372928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:40.373123 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:40.373852 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:40.373866 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:40.374388 (+ 522us) client_negotiation.cc:769] Sending connection context
0809 20:23:40.374723 (+ 335us) client_negotiation.cc:240] Negotiation successful
0809 20:23:40.375193 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":377,"threads_started":1}
I20260809 20:23:40.398399 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.382777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:41195 (local address 127.12.78.3:40593)
0809 20:23:40.383483 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:40.383499 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:40.383613 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:40.384003 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:40.384009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:40.384818 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:40.385273 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.385288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.386270 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.386276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.387276 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.387287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.388035 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.388042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.388209 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.388963 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:40.388994 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:40.390280 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:40.395457 (+ 5177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:40.395460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:40.395463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:40.395885 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:40.396331 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:40.396341 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:40.396346 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:40.396536 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:40.397372 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:40.397376 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:40.397545 (+ 169us) client_negotiation.cc:769] Sending connection context
0809 20:23:40.397761 (+ 216us) client_negotiation.cc:240] Negotiation successful
0809 20:23:40.398088 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":398,"threads_started":1}
I20260809 20:23:40.398731 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.382864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:40593 (local address 127.12.78.62:41195)
0809 20:23:40.383106 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.383116 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.383577 (+ 461us) server_negotiation.cc:407] Connection header received
0809 20:23:40.383740 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.383747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.383846 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.383969 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:40.385436 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.386125 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.387417 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.387839 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.387970 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.390569 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:40.390576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:40.390580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:40.390661 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:40.395312 (+ 4651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:40.396001 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:40.396009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:40.396023 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:40.396190 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:40.396754 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:40.396757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:40.396760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:40.397132 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:40.397298 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.397921 (+ 623us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.398247 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 20:23:40.413592 14819 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:34787
I20260809 20:23:40.413614 14963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:34787 every 8 connection(s)
I20260809 20:23:40.414875 14819 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:23:40.417191 14964 heartbeater.cc:347] Connected to a master server at 127.12.78.62:41195
I20260809 20:23:40.417402 14964 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:40.418016 14964 heartbeater.cc:511] Master 127.12.78.62:41195 requested a full tablet report, sending...
I20260809 20:23:40.419317 14454 ts_manager.cc:194] Registered new tserver with Master: ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787)
I20260809 20:23:40.419874 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 14819
I20260809 20:23:40.420071 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:23:40.420270 14454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:46585
I20260809 20:23:40.420610 12600 external_mini_cluster.cc:1428] Setting key 9aaac0a6f890d9f92a30b83b57d76f9e
I20260809 20:23:40.423527 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 20:23:40 dist-test-slave-rnrw krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 20:23:40 dist-test-slave-rnrw krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:40.478222 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.464085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.12.78.62:41195)
0809 20:23:40.464344 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.464351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.464669 (+ 318us) server_negotiation.cc:407] Connection header received
0809 20:23:40.464891 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.464899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.464971 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.465083 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:40.466508 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.467283 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.468609 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.469042 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.469191 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.473097 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:40.473103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:40.473106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:40.473141 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:40.475503 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:40.476136 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:40.476143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:40.476146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:40.476291 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:40.476732 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:40.476737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:40.476743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:40.477021 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:40.477287 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.477609 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.477863 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260809 20:23:40.482364 14454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:23:40.489281 14454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:23:40.515352 14981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.502159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.12.78.1:35113)
0809 20:23:40.504660 (+ 2501us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.504669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.504701 (+ 32us) server_negotiation.cc:407] Connection header received
0809 20:23:40.504874 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.504886 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.505228 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.506407 (+ 1179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:40.506675 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.508818 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.512597 (+ 3779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.514098 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.514230 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.514448 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.514834 (+ 386us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.515024 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2279,"thread_start_us":2136,"threads_started":1}
I20260809 20:23:40.515384 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.502001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:35113 (local address 127.0.0.1:45350)
0809 20:23:40.503940 (+ 1939us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:40.503959 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:40.504063 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:40.505528 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:40.505540 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:40.505584 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:40.506119 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.506136 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.509798 (+ 3662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.509803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.512411 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.512427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.514314 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.514321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.514561 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.514611 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 20:23:40.514783 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 20:23:40.515003 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1817,"thread_start_us":1678,"threads_started":1}
I20260809 20:23:40.517532 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.502082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.12.78.2:42703)
0809 20:23:40.509410 (+ 7328us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.509420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.509453 (+ 33us) server_negotiation.cc:407] Connection header received
0809 20:23:40.509564 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.509572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.509841 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.510056 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:40.510775 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.512278 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.515130 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.516382 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.516666 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.516871 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.517075 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.517242 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7148,"thread_start_us":7010,"threads_started":1}
I20260809 20:23:40.517639 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.501653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:42703 (local address 127.0.0.1:47122)
0809 20:23:40.503136 (+ 1483us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:40.503185 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:40.503356 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:40.510068 (+ 6712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:40.510073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:40.510100 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:40.510540 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.510553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.512437 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.512442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.514922 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.514936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.516691 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.516698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.516862 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.516897 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 20:23:40.517023 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 20:23:40.517157 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"mutex_wait_us":108,"thread_start_us":1246,"threads_started":1}
I20260809 20:23:40.518515 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.504576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.12.78.3:34787)
0809 20:23:40.509848 (+ 5272us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.509856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.509891 (+ 35us) server_negotiation.cc:407] Connection header received
0809 20:23:40.509985 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.509993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.510216 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.510432 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:40.511196 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.512332 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.515303 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.516773 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.516908 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.517127 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.518016 (+ 889us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.518180 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5086,"thread_start_us":4881,"threads_started":1}
I20260809 20:23:40.519430 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.504084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:34787 (local address 127.0.0.1:57696)
0809 20:23:40.504907 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:40.504925 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:40.505040 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:40.510429 (+ 5389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:40.510433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:40.510462 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:40.510973 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.510984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.512515 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.512519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.515061 (+ 2542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.515075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.517301 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.517308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.517494 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.517528 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 20:23:40.517623 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 20:23:40.517763 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":146,"thread_start_us":548,"threads_started":1}
I20260809 20:23:40.519708 14597 tablet_service.cc:1467] Processing CreateTablet for tablet 4c66f1e312a64ad6abdbdcf88f7e5af7 (DEFAULT_TABLE table=test-table [id=65d013e350e94181ac50246ecb94e121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:40.521343 14597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c66f1e312a64ad6abdbdcf88f7e5af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:40.521277 14744 tablet_service.cc:1467] Processing CreateTablet for tablet 4c66f1e312a64ad6abdbdcf88f7e5af7 (DEFAULT_TABLE table=test-table [id=65d013e350e94181ac50246ecb94e121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:40.522508 14744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c66f1e312a64ad6abdbdcf88f7e5af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:40.522953 14890 tablet_service.cc:1467] Processing CreateTablet for tablet 4c66f1e312a64ad6abdbdcf88f7e5af7 (DEFAULT_TABLE table=test-table [id=65d013e350e94181ac50246ecb94e121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:40.524646 14890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c66f1e312a64ad6abdbdcf88f7e5af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:40.534072 14985 tablet_bootstrap.cc:492] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69: Bootstrap starting.
I20260809 20:23:40.537276 14986 tablet_bootstrap.cc:492] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1: Bootstrap starting.
I20260809 20:23:40.537648 14985 tablet_bootstrap.cc:654] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:40.539040 14985 log.cc:824] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:40.539669 14987 tablet_bootstrap.cc:492] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95: Bootstrap starting.
I20260809 20:23:40.541205 14986 tablet_bootstrap.cc:654] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:40.543226 14987 tablet_bootstrap.cc:654] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:40.543166 14986 log.cc:824] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:40.544634 14987 log.cc:824] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:40.546486 14986 tablet_bootstrap.cc:492] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1: No bootstrap required, opened a new log
I20260809 20:23:40.546802 14986 ts_tablet_manager.cc:1397] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260809 20:23:40.547303 14987 tablet_bootstrap.cc:492] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95: No bootstrap required, opened a new log
I20260809 20:23:40.547597 14987 ts_tablet_manager.cc:1397] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260809 20:23:40.548772 14985 tablet_bootstrap.cc:492] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69: No bootstrap required, opened a new log
I20260809 20:23:40.549077 14985 ts_tablet_manager.cc:1397] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260809 20:23:40.555063 14986 raft_consensus.cc:348] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.555452 14986 raft_consensus.cc:374] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:40.555578 14986 raft_consensus.cc:729] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8264ce544740fbaafde6a9e59659d1, State: Initialized, Role: FOLLOWER
I20260809 20:23:40.556201 14986 consensus_queue.cc:260] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.557775 14986 ts_tablet_manager.cc:1428] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260809 20:23:40.559175 14987 raft_consensus.cc:348] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.559535 14987 raft_consensus.cc:374] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:40.559648 14987 raft_consensus.cc:729] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc20402607e47b39946bf6014a57c95, State: Initialized, Role: FOLLOWER
I20260809 20:23:40.560263 14987 consensus_queue.cc:260] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.560242 14985 raft_consensus.cc:348] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.560678 14985 raft_consensus.cc:374] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:40.560796 14985 raft_consensus.cc:729] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2966bfabe5cf4693bf89c970c3426e69, State: Initialized, Role: FOLLOWER
I20260809 20:23:40.561385 14985 consensus_queue.cc:260] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.562809 14987 ts_tablet_manager.cc:1428] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260809 20:23:40.563012 14964 heartbeater.cc:503] Master 127.12.78.62:41195 was elected leader, sending a full tablet report...
I20260809 20:23:40.564873 14815 heartbeater.cc:503] Master 127.12.78.62:41195 was elected leader, sending a full tablet report...
I20260809 20:23:40.565606 14985 ts_tablet_manager.cc:1428] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260809 20:23:40.573499 14991 raft_consensus.cc:482] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:23:40.573750 14991 raft_consensus.cc:504] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.575304 14991 leader_election.cc:288] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2966bfabe5cf4693bf89c970c3426e69 (127.12.78.2:42703), ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787)
I20260809 20:23:40.584515 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.575636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:35023 (local address 127.12.78.2:42703)
0809 20:23:40.575855 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.575862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.576639 (+ 777us) server_negotiation.cc:407] Connection header received
0809 20:23:40.576996 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.577005 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.577086 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.577153 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:40.577925 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.579014 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.581992 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.583236 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.583361 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.583469 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.584104 (+ 635us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.584245 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 20:23:40.584503 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.575746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:52963 (local address 127.12.78.3:34787)
0809 20:23:40.575923 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.575927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.577365 (+ 1438us) server_negotiation.cc:407] Connection header received
0809 20:23:40.577525 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.577528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.577586 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.577659 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:40.578262 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.579285 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.581983 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.583215 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.583360 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.583469 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.584104 (+ 635us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.584240 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 20:23:40.584818 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.575750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:42703 (local address 127.12.78.1:35023)
0809 20:23:40.576502 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:40.576525 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:40.576779 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:40.577381 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:40.577388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:40.577408 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:40.577776 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.577786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.579131 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.579135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.581764 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.581775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.583519 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.583529 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.583810 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.583977 (+ 167us) client_negotiation.cc:769] Sending connection context
0809 20:23:40.584064 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 20:23:40.584251 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":272,"thread_start_us":798,"threads_started":1}
I20260809 20:23:40.585192 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.576259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:34787 (local address 127.12.78.1:52963)
0809 20:23:40.577265 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:40.577290 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:40.577402 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:40.577745 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:40.577755 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:40.577779 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:40.578115 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.578123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.579425 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.579432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.581760 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.581775 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.583561 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.583567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.583807 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.583955 (+ 148us) client_negotiation.cc:769] Sending connection context
0809 20:23:40.584065 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 20:23:40.584251 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":158,"thread_start_us":584,"threads_started":1}
W20260809 20:23:40.585721 14674 tablet.cc:2369] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:23:40.586169 14770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c66f1e312a64ad6abdbdcf88f7e5af7" candidate_uuid: "ff8264ce544740fbaafde6a9e59659d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2966bfabe5cf4693bf89c970c3426e69" is_pre_election: true
I20260809 20:23:40.586721 14770 raft_consensus.cc:2454] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff8264ce544740fbaafde6a9e59659d1 in term 0.
I20260809 20:23:40.586793 14917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c66f1e312a64ad6abdbdcf88f7e5af7" candidate_uuid: "ff8264ce544740fbaafde6a9e59659d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc20402607e47b39946bf6014a57c95" is_pre_election: true
I20260809 20:23:40.587349 14917 raft_consensus.cc:2454] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff8264ce544740fbaafde6a9e59659d1 in term 0.
I20260809 20:23:40.587463 14548 leader_election.cc:302] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2966bfabe5cf4693bf89c970c3426e69, ff8264ce544740fbaafde6a9e59659d1; no voters:
I20260809 20:23:40.587911 14991 raft_consensus.cc:2781] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:23:40.588085 14991 raft_consensus.cc:482] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:23:40.588200 14991 raft_consensus.cc:3037] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:40.590586 14991 raft_consensus.cc:504] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.591216 14991 leader_election.cc:288] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [CANDIDATE]: Term 1 election: Requested vote from peers 2966bfabe5cf4693bf89c970c3426e69 (127.12.78.2:42703), ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787)
I20260809 20:23:40.591732 14917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c66f1e312a64ad6abdbdcf88f7e5af7" candidate_uuid: "ff8264ce544740fbaafde6a9e59659d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc20402607e47b39946bf6014a57c95"
I20260809 20:23:40.591768 14770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c66f1e312a64ad6abdbdcf88f7e5af7" candidate_uuid: "ff8264ce544740fbaafde6a9e59659d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2966bfabe5cf4693bf89c970c3426e69"
I20260809 20:23:40.591933 14917 raft_consensus.cc:3037] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:40.591976 14770 raft_consensus.cc:3037] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:40.594269 14917 raft_consensus.cc:2454] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff8264ce544740fbaafde6a9e59659d1 in term 1.
I20260809 20:23:40.594276 14770 raft_consensus.cc:2454] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff8264ce544740fbaafde6a9e59659d1 in term 1.
I20260809 20:23:40.594743 14546 leader_election.cc:302] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ebc20402607e47b39946bf6014a57c95, ff8264ce544740fbaafde6a9e59659d1; no voters:
I20260809 20:23:40.595082 14991 raft_consensus.cc:2781] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:40.595505 14991 raft_consensus.cc:686] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [term 1 LEADER]: Becoming Leader. State: Replica: ff8264ce544740fbaafde6a9e59659d1, State: Running, Role: LEADER
I20260809 20:23:40.596083 14991 consensus_queue.cc:237] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.601058 14455 catalog_manager.cc:5495] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff8264ce544740fbaafde6a9e59659d1 (127.12.78.1). New cstate: current_term: 1 leader_uuid: "ff8264ce544740fbaafde6a9e59659d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } health_report { overall_health: UNKNOWN } } }
W20260809 20:23:40.631891 14816 tablet.cc:2369] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 20:23:40 dist-test-slave-rnrw krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307018, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
W20260809 20:23:40.666463 14965 tablet.cc:2369] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:23:40.669026 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.655006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.12.78.62:41195)
0809 20:23:40.655190 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.655198 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.655868 (+ 670us) server_negotiation.cc:407] Connection header received
0809 20:23:40.656097 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.656101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.656159 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.656283 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:40.657546 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.658281 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.659531 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.659998 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.660154 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.663644 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:40.663650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:40.663653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:40.663682 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:40.666138 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:40.666926 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:40.666934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:40.666937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:40.667075 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:40.667461 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:40.667469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:40.667472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:40.667747 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:40.667910 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.668329 (+ 419us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.668716 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":60}
I20260809 20:23:40.669035 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.655017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:41195 (local address 127.0.0.1:41814)
0809 20:23:40.655621 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:40.655641 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:40.655956 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:40.656300 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:40.656304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:40.657108 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:40.657388 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.657398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.658408 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.658417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.659391 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.659403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.660195 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.660201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.660380 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.661142 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:40.661159 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:40.663452 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:40.666288 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:40.666303 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:40.666311 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:40.666812 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:40.667194 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:40.667199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:40.667202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:40.667339 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:40.667885 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:40.667892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:40.668052 (+ 160us) client_negotiation.cc:769] Sending connection context
0809 20:23:40.668224 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 20:23:40.668524 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":349,"threads_started":1}
I20260809 20:23:40.674508 14455 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:41814:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 20:23:40.675506 14455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:23:40.682535 14890 tablet_service.cc:1467] Processing CreateTablet for tablet c3e4a6d562024391923e303d981b513f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7ff21eda6ba40f4b5c16d14fe8d2d47]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 20:23:40.682468 14597 tablet_service.cc:1467] Processing CreateTablet for tablet c3e4a6d562024391923e303d981b513f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7ff21eda6ba40f4b5c16d14fe8d2d47]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 20:23:40.683386 14597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e4a6d562024391923e303d981b513f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:40.683393 14890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e4a6d562024391923e303d981b513f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:40.683732 14744 tablet_service.cc:1467] Processing CreateTablet for tablet c3e4a6d562024391923e303d981b513f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7ff21eda6ba40f4b5c16d14fe8d2d47]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 20:23:40.684533 14744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e4a6d562024391923e303d981b513f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:40.690629 14987 tablet_bootstrap.cc:492] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95: Bootstrap starting.
I20260809 20:23:40.690647 14986 tablet_bootstrap.cc:492] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1: Bootstrap starting.
I20260809 20:23:40.692044 14985 tablet_bootstrap.cc:492] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69: Bootstrap starting.
I20260809 20:23:40.692306 14987 tablet_bootstrap.cc:654] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:40.692306 14986 tablet_bootstrap.cc:654] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:40.694039 14985 tablet_bootstrap.cc:654] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:40.704681 14985 tablet_bootstrap.cc:492] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69: No bootstrap required, opened a new log
I20260809 20:23:40.704994 14985 ts_tablet_manager.cc:1397] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260809 20:23:40.705288 14986 tablet_bootstrap.cc:492] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1: No bootstrap required, opened a new log
I20260809 20:23:40.705447 14986 ts_tablet_manager.cc:1397] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260809 20:23:40.706213 14985 raft_consensus.cc:348] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.706277 14986 raft_consensus.cc:348] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.706517 14986 raft_consensus.cc:374] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:40.706611 14986 raft_consensus.cc:729] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8264ce544740fbaafde6a9e59659d1, State: Initialized, Role: FOLLOWER
I20260809 20:23:40.706640 14985 raft_consensus.cc:374] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:40.706835 14985 raft_consensus.cc:729] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2966bfabe5cf4693bf89c970c3426e69, State: Initialized, Role: FOLLOWER
I20260809 20:23:40.706877 14986 consensus_queue.cc:260] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.707044 14987 tablet_bootstrap.cc:492] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95: No bootstrap required, opened a new log
I20260809 20:23:40.707262 14985 consensus_queue.cc:260] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.707486 14986 ts_tablet_manager.cc:1428] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260809 20:23:40.707357 14991 tablet_replica.cc:439] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } } }
I20260809 20:23:40.707705 14991 tablet_replica.cc:442] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:40.708015 14987 ts_tablet_manager.cc:1397] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20260809 20:23:40.707931 14993 tablet_replica.cc:439] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } } }
I20260809 20:23:40.708300 14993 tablet_replica.cc:442] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:40.708938 14987 raft_consensus.cc:348] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.709199 14987 raft_consensus.cc:374] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:40.709293 14987 raft_consensus.cc:729] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc20402607e47b39946bf6014a57c95, State: Initialized, Role: FOLLOWER
I20260809 20:23:40.709519 14987 consensus_queue.cc:260] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.709666 14985 ts_tablet_manager.cc:1428] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 20:23:40.709986 14992 tablet_replica.cc:439] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } } }
I20260809 20:23:40.710373 14992 tablet_replica.cc:442] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:40.710821 14987 ts_tablet_manager.cc:1428] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 20:23:40.756247 14993 raft_consensus.cc:482] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:23:40.756496 14993 raft_consensus.cc:504] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.758064 14993 leader_election.cc:288] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8264ce544740fbaafde6a9e59659d1 (127.12.78.1:35113), ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787)
I20260809 20:23:40.765508 14981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.758444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:53193 (local address 127.12.78.1:35113)
0809 20:23:40.758728 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.758738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.759182 (+ 444us) server_negotiation.cc:407] Connection header received
0809 20:23:40.759444 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.759450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.759542 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.759630 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:40.760460 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.761540 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.763550 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.764495 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.764674 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.764752 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.765136 (+ 384us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.765266 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260809 20:23:40.765621 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.758432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:35113 (local address 127.12.78.2:53193)
0809 20:23:40.759068 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:40.759085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:40.759254 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:40.759868 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:40.759872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:40.759894 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:40.760286 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.760299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.761679 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.761685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.763373 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.763385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.764766 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.764772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.764950 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.764997 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 20:23:40.765090 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 20:23:40.765242 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":373,"threads_started":1}
I20260809 20:23:40.765961 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.759389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:35955 (local address 127.12.78.3:34787)
0809 20:23:40.759769 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.759777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.760094 (+ 317us) server_negotiation.cc:407] Connection header received
0809 20:23:40.760263 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.760270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.760368 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.760469 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:23:40.761133 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.762095 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.763977 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.764941 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.765090 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.765178 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.765535 (+ 357us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.765683 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260809 20:23:40.766027 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.759263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:34787 (local address 127.12.78.2:35955)
0809 20:23:40.759974 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:40.759992 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:40.760103 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:40.760494 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:40.760498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:40.760525 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:23:40.760964 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.760976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.762246 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.762250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.763821 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.763835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.765182 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.765189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.765359 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.765393 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 20:23:40.765489 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 20:23:40.765635 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":1,"thread_start_us":418,"threads_started":1}
I20260809 20:23:40.766624 14917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e4a6d562024391923e303d981b513f" candidate_uuid: "2966bfabe5cf4693bf89c970c3426e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc20402607e47b39946bf6014a57c95" is_pre_election: true
I20260809 20:23:40.766705 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e4a6d562024391923e303d981b513f" candidate_uuid: "2966bfabe5cf4693bf89c970c3426e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8264ce544740fbaafde6a9e59659d1" is_pre_election: true
I20260809 20:23:40.766942 14917 raft_consensus.cc:2454] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2966bfabe5cf4693bf89c970c3426e69 in term 0.
I20260809 20:23:40.767144 14619 raft_consensus.cc:2454] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2966bfabe5cf4693bf89c970c3426e69 in term 0.
I20260809 20:23:40.767414 14696 leader_election.cc:302] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2966bfabe5cf4693bf89c970c3426e69, ebc20402607e47b39946bf6014a57c95; no voters:
I20260809 20:23:40.767819 14993 raft_consensus.cc:2781] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:23:40.767967 14993 raft_consensus.cc:482] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:23:40.768087 14993 raft_consensus.cc:3037] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:40.770309 14993 raft_consensus.cc:504] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.770860 14993 leader_election.cc:288] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [CANDIDATE]: Term 1 election: Requested vote from peers ff8264ce544740fbaafde6a9e59659d1 (127.12.78.1:35113), ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787)
I20260809 20:23:40.771378 14917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e4a6d562024391923e303d981b513f" candidate_uuid: "2966bfabe5cf4693bf89c970c3426e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc20402607e47b39946bf6014a57c95"
I20260809 20:23:40.771415 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e4a6d562024391923e303d981b513f" candidate_uuid: "2966bfabe5cf4693bf89c970c3426e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8264ce544740fbaafde6a9e59659d1"
I20260809 20:23:40.771595 14917 raft_consensus.cc:3037] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:40.771612 14619 raft_consensus.cc:3037] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:40.773746 14917 raft_consensus.cc:2454] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2966bfabe5cf4693bf89c970c3426e69 in term 1.
I20260809 20:23:40.773931 14619 raft_consensus.cc:2454] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2966bfabe5cf4693bf89c970c3426e69 in term 1.
I20260809 20:23:40.774165 14696 leader_election.cc:302] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2966bfabe5cf4693bf89c970c3426e69, ebc20402607e47b39946bf6014a57c95; no voters:
I20260809 20:23:40.774459 14993 raft_consensus.cc:2781] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:40.774909 14993 raft_consensus.cc:686] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [term 1 LEADER]: Becoming Leader. State: Replica: 2966bfabe5cf4693bf89c970c3426e69, State: Running, Role: LEADER
I20260809 20:23:40.775388 14993 consensus_queue.cc:237] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:40.779415 15011 tablet_replica.cc:439] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69: TxnStatusTablet state changed. Reason: New leader 2966bfabe5cf4693bf89c970c3426e69. Latest consensus state: current_term: 1 leader_uuid: "2966bfabe5cf4693bf89c970c3426e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } } }
I20260809 20:23:40.779928 15011 tablet_replica.cc:442] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69: This TxnStatusTablet replica's current role is: LEADER
I20260809 20:23:40.781162 15013 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 20:23:40.781427 15013 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 20:23:40.781878 14455 catalog_manager.cc:5495] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2966bfabe5cf4693bf89c970c3426e69 (127.12.78.2). New cstate: current_term: 1 leader_uuid: "2966bfabe5cf4693bf89c970c3426e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } health_report { overall_health: UNKNOWN } } }
I20260809 20:23:40.817163 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.811110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.12.78.2:42703)
0809 20:23:40.811280 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.811284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.811302 (+ 18us) server_negotiation.cc:407] Connection header received
0809 20:23:40.811555 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.811558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.811642 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.811766 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:23:40.812204 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.813082 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.814970 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.815378 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.815466 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.816015 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:23:40.816506 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:23:40.816743 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.816860 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.816925 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 20:23:40.817282 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.810941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:42703 (local address 127.0.0.1:47130)
0809 20:23:40.811195 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:40.811218 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:40.811359 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:40.811840 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:40.811845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:40.811859 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 20:23:40.812060 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.812074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.813215 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.813221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.814837 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.814853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.815539 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.815546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.815768 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.815841 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 20:23:40.816709 (+ 868us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 20:23:40.816732 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 20:23:40.816831 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 20:23:40.816988 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 20:23:40.825196 14619 raft_consensus.cc:1261] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Refusing update from remote peer 2966bfabe5cf4693bf89c970c3426e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:40.825245 14917 raft_consensus.cc:1261] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Refusing update from remote peer 2966bfabe5cf4693bf89c970c3426e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:40.826160 14993 consensus_queue.cc:1035] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:40.826558 15011 consensus_queue.cc:1035] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:40.841033 14991 tablet_replica.cc:439] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1: TxnStatusTablet state changed. Reason: New leader 2966bfabe5cf4693bf89c970c3426e69. Latest consensus state: current_term: 1 leader_uuid: "2966bfabe5cf4693bf89c970c3426e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } } }
I20260809 20:23:40.841390 14991 tablet_replica.cc:442] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:40.842458 15016 mvcc.cc:205] Tried to move back new op lower bound from 7316713557289263104 to 7316713557100265472. Current Snapshot: MvccSnapshot[applied={T|T < 7316713557289263104}]
I20260809 20:23:40.843789 14993 tablet_replica.cc:439] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2966bfabe5cf4693bf89c970c3426e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } } }
I20260809 20:23:40.844007 15019 mvcc.cc:205] Tried to move back new op lower bound from 7316713557289263104 to 7316713557100265472. Current Snapshot: MvccSnapshot[applied={T|T < 7316713557289263104}]
I20260809 20:23:40.844067 14993 tablet_replica.cc:442] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69: This TxnStatusTablet replica's current role is: LEADER
I20260809 20:23:40.848459 15011 tablet_replica.cc:439] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2966bfabe5cf4693bf89c970c3426e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } } }
I20260809 20:23:40.848816 15011 tablet_replica.cc:442] T c3e4a6d562024391923e303d981b513f P 2966bfabe5cf4693bf89c970c3426e69: This TxnStatusTablet replica's current role is: LEADER
I20260809 20:23:40.849920 14992 tablet_replica.cc:439] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95: TxnStatusTablet state changed. Reason: New leader 2966bfabe5cf4693bf89c970c3426e69. Latest consensus state: current_term: 1 leader_uuid: "2966bfabe5cf4693bf89c970c3426e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } } }
I20260809 20:23:40.850210 14992 tablet_replica.cc:442] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:40.854676 14991 tablet_replica.cc:439] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2966bfabe5cf4693bf89c970c3426e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } } }
I20260809 20:23:40.854954 14991 tablet_replica.cc:442] T c3e4a6d562024391923e303d981b513f P ff8264ce544740fbaafde6a9e59659d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:40.858019 15017 mvcc.cc:205] Tried to move back new op lower bound from 7316713557289263104 to 7316713557100265472. Current Snapshot: MvccSnapshot[applied={T|T < 7316713557289263104}]
I20260809 20:23:40.860809 14992 tablet_replica.cc:439] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2966bfabe5cf4693bf89c970c3426e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } } }
I20260809 20:23:40.861110 14992 tablet_replica.cc:442] T c3e4a6d562024391923e303d981b513f P ebc20402607e47b39946bf6014a57c95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:23:40.869159 14981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.863820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.12.78.1:35113)
0809 20:23:40.863998 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.864004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.864022 (+ 18us) server_negotiation.cc:407] Connection header received
0809 20:23:40.864116 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.864120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.864247 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.864446 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:23:40.864855 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.865584 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.866904 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.867329 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.867488 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.867949 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:23:40.868474 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:23:40.868714 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.868770 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.868854 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 20:23:40.888928 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.881112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:46121 (local address 127.12.78.2:42703)
0809 20:23:40.881301 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.881307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.882034 (+ 727us) server_negotiation.cc:407] Connection header received
0809 20:23:40.882259 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.882266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.882380 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.882505 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:23:40.883115 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.884369 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.886503 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.887318 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.887507 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.888107 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:23:40.888251 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:23:40.888344 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.888507 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.888622 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 20:23:40.889266 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.881141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:42703 (local address 127.12.78.1:46121)
0809 20:23:40.881897 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:40.881919 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:40.882072 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:40.882531 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:40.882536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:40.882550 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 20:23:40.882922 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.882934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.884594 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.884601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.886344 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.886359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.887597 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.887604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.887790 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.887863 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 20:23:40.888362 (+ 499us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 20:23:40.888375 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 20:23:40.888460 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 20:23:40.888646 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":368,"threads_started":1}
I20260809 20:23:40.901535 14917 raft_consensus.cc:1261] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Refusing update from remote peer ff8264ce544740fbaafde6a9e59659d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:40.902194 14770 raft_consensus.cc:1261] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Refusing update from remote peer ff8264ce544740fbaafde6a9e59659d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:40.903137 14991 consensus_queue.cc:1035] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:40.903470 14997 consensus_queue.cc:1035] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:40.952637 14981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.945420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:56945 (local address 127.12.78.1:35113)
0809 20:23:40.945609 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:40.945618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:40.946418 (+ 800us) server_negotiation.cc:407] Connection header received
0809 20:23:40.946643 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:40.946648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:40.946770 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:40.946863 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:23:40.947757 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.948484 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.950574 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.951162 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.951297 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.951756 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:23:40.951894 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:23:40.951977 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:40.952222 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 20:23:40.952309 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 20:23:40.953003 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:40.945422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:35113 (local address 127.12.78.2:56945)
0809 20:23:40.946232 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:40.946264 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:40.946500 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:40.946912 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:40.946917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:40.946931 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 20:23:40.947354 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.947366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.948646 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.948651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.950435 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:40.950450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:40.951341 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:40.951349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:40.951498 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:40.951551 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 20:23:40.952025 (+ 474us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 20:23:40.952035 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 20:23:40.952190 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 20:23:40.952328 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":397,"threads_started":1}
I20260809 20:23:40.991983 14455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:23:40.992327 14455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:23:40.996766 14455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39: Sending DeleteTablet for 3 replicas of tablet 4c66f1e312a64ad6abdbdcf88f7e5af7
I20260809 20:23:40.998338 14597 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c66f1e312a64ad6abdbdcf88f7e5af7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:40 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:45350
I20260809 20:23:40.998817 14890 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c66f1e312a64ad6abdbdcf88f7e5af7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:40 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:57696
I20260809 20:23:40.999768 14744 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c66f1e312a64ad6abdbdcf88f7e5af7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:40 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:47122
I20260809 20:23:41.004127 14455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:23:41.006309 14455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:23:41.013573 15036 tablet_replica.cc:330] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1: stopping tablet replica
I20260809 20:23:41.014084 15036 raft_consensus.cc:2229] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:23:41.014814 15036 raft_consensus.cc:2258] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:41.017930 15036 ts_tablet_manager.cc:1905] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:41.019846 14597 tablet_service.cc:1467] Processing CreateTablet for tablet a5d156cc6e654e26b23489ee977067bf (DEFAULT_TABLE table=test-table [id=17f5f99f0f03442db3f3bddbcd1e978d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:41.020318 14597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5d156cc6e654e26b23489ee977067bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:41.020563 14743 tablet_service.cc:1467] Processing CreateTablet for tablet a5d156cc6e654e26b23489ee977067bf (DEFAULT_TABLE table=test-table [id=17f5f99f0f03442db3f3bddbcd1e978d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:41.021180 14743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5d156cc6e654e26b23489ee977067bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:41.025787 14889 tablet_service.cc:1467] Processing CreateTablet for tablet a5d156cc6e654e26b23489ee977067bf (DEFAULT_TABLE table=test-table [id=17f5f99f0f03442db3f3bddbcd1e978d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:41.026309 14889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5d156cc6e654e26b23489ee977067bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:41.027539 14986 tablet_bootstrap.cc:492] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1: Bootstrap starting.
I20260809 20:23:41.029364 14986 tablet_bootstrap.cc:654] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:41.035800 14987 tablet_bootstrap.cc:492] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95: Bootstrap starting.
I20260809 20:23:41.037835 14987 tablet_bootstrap.cc:654] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:41.047499 14985 tablet_bootstrap.cc:492] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69: Bootstrap starting.
I20260809 20:23:41.049310 14986 tablet_bootstrap.cc:492] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1: No bootstrap required, opened a new log
I20260809 20:23:41.049544 14986 ts_tablet_manager.cc:1397] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.010s
I20260809 20:23:41.049808 14985 tablet_bootstrap.cc:654] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:41.050387 14986 raft_consensus.cc:348] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:41.050642 14986 raft_consensus.cc:374] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:41.050724 14986 raft_consensus.cc:729] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8264ce544740fbaafde6a9e59659d1, State: Initialized, Role: FOLLOWER
I20260809 20:23:41.050983 14986 consensus_queue.cc:260] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:41.054052 14986 ts_tablet_manager.cc:1428] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 20:23:41.057960 15036 ts_tablet_manager.cc:1918] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:23:41.058210 15036 log.cc:1196] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/4c66f1e312a64ad6abdbdcf88f7e5af7
I20260809 20:23:41.058887 15036 ts_tablet_manager.cc:1939] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ff8264ce544740fbaafde6a9e59659d1: Deleting consensus metadata
I20260809 20:23:41.060930 15038 tablet_replica.cc:330] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69: stopping tablet replica
I20260809 20:23:41.061177 14439 catalog_manager.cc:4841] TS ff8264ce544740fbaafde6a9e59659d1 (127.12.78.1:35113): tablet 4c66f1e312a64ad6abdbdcf88f7e5af7 (table test-table [id=65d013e350e94181ac50246ecb94e121]) successfully deleted
I20260809 20:23:41.061918 15037 tablet_replica.cc:330] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95: stopping tablet replica
I20260809 20:23:41.062575 15037 raft_consensus.cc:2229] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:23:41.063057 15037 raft_consensus.cc:2258] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:41.063941 15038 raft_consensus.cc:2229] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:23:41.064424 15038 raft_consensus.cc:2258] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:41.065312 15037 ts_tablet_manager.cc:1905] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:41.068318 15038 ts_tablet_manager.cc:1905] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:41.069707 14987 tablet_bootstrap.cc:492] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95: No bootstrap required, opened a new log
I20260809 20:23:41.069895 14987 ts_tablet_manager.cc:1397] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.008s
I20260809 20:23:41.070815 14987 raft_consensus.cc:348] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:41.071080 14987 raft_consensus.cc:374] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:41.071169 14987 raft_consensus.cc:729] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc20402607e47b39946bf6014a57c95, State: Initialized, Role: FOLLOWER
I20260809 20:23:41.071507 14987 consensus_queue.cc:260] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:41.072250 14987 ts_tablet_manager.cc:1428] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260809 20:23:41.074172 15037 ts_tablet_manager.cc:1918] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:23:41.074411 15037 log.cc:1196] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/4c66f1e312a64ad6abdbdcf88f7e5af7
I20260809 20:23:41.074707 14985 tablet_bootstrap.cc:492] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69: No bootstrap required, opened a new log
I20260809 20:23:41.074896 14985 ts_tablet_manager.cc:1397] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.008s
I20260809 20:23:41.075176 15037 ts_tablet_manager.cc:1939] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P ebc20402607e47b39946bf6014a57c95: Deleting consensus metadata
I20260809 20:23:41.075738 14985 raft_consensus.cc:348] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:41.075984 14985 raft_consensus.cc:374] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:41.076069 14985 raft_consensus.cc:729] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2966bfabe5cf4693bf89c970c3426e69, State: Initialized, Role: FOLLOWER
I20260809 20:23:41.076324 14985 consensus_queue.cc:260] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:41.076956 14985 ts_tablet_manager.cc:1428] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 20:23:41.077387 14439 catalog_manager.cc:4841] TS ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787): tablet 4c66f1e312a64ad6abdbdcf88f7e5af7 (table test-table [id=65d013e350e94181ac50246ecb94e121]) successfully deleted
I20260809 20:23:41.088706 15038 ts_tablet_manager.cc:1918] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:23:41.088925 15038 log.cc:1196] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/4c66f1e312a64ad6abdbdcf88f7e5af7
I20260809 20:23:41.089516 15038 ts_tablet_manager.cc:1939] T 4c66f1e312a64ad6abdbdcf88f7e5af7 P 2966bfabe5cf4693bf89c970c3426e69: Deleting consensus metadata
I20260809 20:23:41.094556 14441 catalog_manager.cc:4841] TS 2966bfabe5cf4693bf89c970c3426e69 (127.12.78.2:42703): tablet 4c66f1e312a64ad6abdbdcf88f7e5af7 (table test-table [id=65d013e350e94181ac50246ecb94e121]) successfully deleted
I20260809 20:23:41.263518 15029 raft_consensus.cc:482] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:23:41.263808 15029 raft_consensus.cc:504] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:41.264777 15029 leader_election.cc:288] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8264ce544740fbaafde6a9e59659d1 (127.12.78.1:35113), ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787)
I20260809 20:23:41.265480 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d156cc6e654e26b23489ee977067bf" candidate_uuid: "2966bfabe5cf4693bf89c970c3426e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8264ce544740fbaafde6a9e59659d1" is_pre_election: true
I20260809 20:23:41.265513 14916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d156cc6e654e26b23489ee977067bf" candidate_uuid: "2966bfabe5cf4693bf89c970c3426e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc20402607e47b39946bf6014a57c95" is_pre_election: true
I20260809 20:23:41.265779 14619 raft_consensus.cc:2454] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2966bfabe5cf4693bf89c970c3426e69 in term 0.
I20260809 20:23:41.265823 14916 raft_consensus.cc:2454] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2966bfabe5cf4693bf89c970c3426e69 in term 0.
I20260809 20:23:41.266378 14696 leader_election.cc:302] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2966bfabe5cf4693bf89c970c3426e69, ff8264ce544740fbaafde6a9e59659d1; no voters:
I20260809 20:23:41.267239 15029 raft_consensus.cc:2781] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:23:41.267360 15029 raft_consensus.cc:482] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:23:41.267493 15029 raft_consensus.cc:3037] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:41.270840 15029 raft_consensus.cc:504] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:41.271622 15029 leader_election.cc:288] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [CANDIDATE]: Term 1 election: Requested vote from peers ff8264ce544740fbaafde6a9e59659d1 (127.12.78.1:35113), ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787)
I20260809 20:23:41.272300 14916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d156cc6e654e26b23489ee977067bf" candidate_uuid: "2966bfabe5cf4693bf89c970c3426e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc20402607e47b39946bf6014a57c95"
I20260809 20:23:41.272356 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d156cc6e654e26b23489ee977067bf" candidate_uuid: "2966bfabe5cf4693bf89c970c3426e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8264ce544740fbaafde6a9e59659d1"
I20260809 20:23:41.272567 14916 raft_consensus.cc:3037] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:41.272692 14619 raft_consensus.cc:3037] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:41.275658 14916 raft_consensus.cc:2454] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2966bfabe5cf4693bf89c970c3426e69 in term 1.
I20260809 20:23:41.275838 14619 raft_consensus.cc:2454] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2966bfabe5cf4693bf89c970c3426e69 in term 1.
I20260809 20:23:41.276369 14696 leader_election.cc:302] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2966bfabe5cf4693bf89c970c3426e69, ff8264ce544740fbaafde6a9e59659d1; no voters:
I20260809 20:23:41.276772 15029 raft_consensus.cc:2781] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:41.276978 15029 raft_consensus.cc:686] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [term 1 LEADER]: Becoming Leader. State: Replica: 2966bfabe5cf4693bf89c970c3426e69, State: Running, Role: LEADER
I20260809 20:23:41.277308 15029 consensus_queue.cc:237] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:41.280360 14455 catalog_manager.cc:5495] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2966bfabe5cf4693bf89c970c3426e69 (127.12.78.2). New cstate: current_term: 1 leader_uuid: "2966bfabe5cf4693bf89c970c3426e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } health_report { overall_health: UNKNOWN } } }
I20260809 20:23:41.360625 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:41.355501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.12.78.2:42703)
0809 20:23:41.355680 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:41.355687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:41.356006 (+ 319us) server_negotiation.cc:407] Connection header received
0809 20:23:41.356211 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:41.356217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:41.356297 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:41.356391 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:23:41.356843 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:41.357540 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:41.358849 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:41.359261 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:41.359398 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:41.359819 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:23:41.359922 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:23:41.360052 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:41.360195 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 20:23:41.360277 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 20:23:41.373910 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:41.368652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:45599 (local address 127.12.78.2:42703)
0809 20:23:41.368848 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:41.368854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:41.369011 (+ 157us) server_negotiation.cc:407] Connection header received
0809 20:23:41.369243 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:41.369247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:41.369312 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:41.369440 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:23:41.369799 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:41.370579 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:41.371861 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:41.372441 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:41.372612 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:41.373051 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:23:41.373186 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:23:41.373319 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:41.373459 (+ 140us) server_negotiation.cc:299] Negotiation successful
0809 20:23:41.373529 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":59}
I20260809 20:23:41.373986 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:41.368652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:42703 (local address 127.12.78.2:45599)
0809 20:23:41.368875 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:41.368900 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:41.369081 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:41.369451 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:41.369456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:41.369466 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 20:23:41.369649 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:41.369658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:41.370719 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:41.370725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:41.371711 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:41.371722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:41.372627 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:41.372632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:41.372818 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:41.372853 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 20:23:41.373352 (+ 499us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 20:23:41.373358 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 20:23:41.373440 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 20:23:41.373575 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":47}
I20260809 20:23:41.384322 14618 raft_consensus.cc:1261] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Refusing update from remote peer 2966bfabe5cf4693bf89c970c3426e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:41.384528 14916 raft_consensus.cc:1261] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Refusing update from remote peer 2966bfabe5cf4693bf89c970c3426e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:41.385047 15029 consensus_queue.cc:1035] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:41.385625 15029 consensus_queue.cc:1035] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 20:23:41.411933 15016 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 20:23:43.033797 14454 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:23:43.034200 14454 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:23:43.039587 14454 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39: Sending DeleteTablet for 3 replicas of tablet a5d156cc6e654e26b23489ee977067bf
I20260809 20:23:43.040510 14743 tablet_service.cc:1514] Processing DeleteTablet for tablet a5d156cc6e654e26b23489ee977067bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:43 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:47122
I20260809 20:23:43.040534 14597 tablet_service.cc:1514] Processing DeleteTablet for tablet a5d156cc6e654e26b23489ee977067bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:43 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:45350
I20260809 20:23:43.040689 14890 tablet_service.cc:1514] Processing DeleteTablet for tablet a5d156cc6e654e26b23489ee977067bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:43 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:57696
I20260809 20:23:43.051810 15068 tablet_replica.cc:330] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95: stopping tablet replica
I20260809 20:23:43.052265 15068 raft_consensus.cc:2229] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:23:43.052700 15068 raft_consensus.cc:2258] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:43.053180 15066 tablet_replica.cc:330] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69: stopping tablet replica
I20260809 20:23:43.053524 15066 raft_consensus.cc:2229] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:23:43.053958 15068 ts_tablet_manager.cc:1905] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:43.054291 15066 raft_consensus.cc:2258] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:43.055248 15066 ts_tablet_manager.cc:1905] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:43.061888 15068 ts_tablet_manager.cc:1918] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 20:23:43.062306 15067 tablet_replica.cc:330] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1: stopping tablet replica
I20260809 20:23:43.062520 15067 raft_consensus.cc:2229] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:23:43.062605 15068 log.cc:1196] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/a5d156cc6e654e26b23489ee977067bf
I20260809 20:23:43.062959 15067 raft_consensus.cc:2258] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:43.063189 15068 ts_tablet_manager.cc:1939] T a5d156cc6e654e26b23489ee977067bf P ebc20402607e47b39946bf6014a57c95: Deleting consensus metadata
I20260809 20:23:43.063539 15066 ts_tablet_manager.cc:1918] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 20:23:43.063732 15066 log.cc:1196] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/a5d156cc6e654e26b23489ee977067bf
I20260809 20:23:43.064278 15066 ts_tablet_manager.cc:1939] T a5d156cc6e654e26b23489ee977067bf P 2966bfabe5cf4693bf89c970c3426e69: Deleting consensus metadata
I20260809 20:23:43.064306 15067 ts_tablet_manager.cc:1905] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:43.064543 14439 catalog_manager.cc:4841] TS ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787): tablet a5d156cc6e654e26b23489ee977067bf (table test-table [id=17f5f99f0f03442db3f3bddbcd1e978d]) successfully deleted
I20260809 20:23:43.065608 14441 catalog_manager.cc:4841] TS 2966bfabe5cf4693bf89c970c3426e69 (127.12.78.2:42703): tablet a5d156cc6e654e26b23489ee977067bf (table test-table [id=17f5f99f0f03442db3f3bddbcd1e978d]) successfully deleted
I20260809 20:23:43.070073 15067 ts_tablet_manager.cc:1918] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 20:23:43.070266 15067 log.cc:1196] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/a5d156cc6e654e26b23489ee977067bf
I20260809 20:23:43.070775 15067 ts_tablet_manager.cc:1939] T a5d156cc6e654e26b23489ee977067bf P ff8264ce544740fbaafde6a9e59659d1: Deleting consensus metadata
I20260809 20:23:43.071897 14439 catalog_manager.cc:4841] TS ff8264ce544740fbaafde6a9e59659d1 (127.12.78.1:35113): tablet a5d156cc6e654e26b23489ee977067bf (table test-table [id=17f5f99f0f03442db3f3bddbcd1e978d]) successfully deleted
Aug 09 20:23:43 dist-test-slave-rnrw krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 20:23:43 dist-test-slave-rnrw krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:43.118234 15077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:43.098385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.12.78.62:41195)
0809 20:23:43.099593 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:43.099597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:43.099616 (+ 19us) server_negotiation.cc:407] Connection header received
0809 20:23:43.099695 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:43.099701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:43.099788 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:43.099997 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:43.101661 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:43.102633 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:43.103853 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:43.104264 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:43.104404 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:43.112896 (+ 8492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:43.112901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:43.112904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:43.112939 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:43.115547 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:43.116193 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:43.116200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:43.116208 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:43.116333 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:43.116778 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:43.116784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:43.116793 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:43.117069 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:43.117333 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:43.117639 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 20:23:43.117878 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096,"thread_start_us":937,"threads_started":1}
I20260809 20:23:43.121490 14454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:23:43.122109 14454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:23:43.128396 14597 tablet_service.cc:1467] Processing CreateTablet for tablet b849e9505f5049ffa7fe153d355c3953 (DEFAULT_TABLE table=test-table [id=e5eea92fbfc74ad2b915c81df03779a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:43.128479 14743 tablet_service.cc:1467] Processing CreateTablet for tablet b849e9505f5049ffa7fe153d355c3953 (DEFAULT_TABLE table=test-table [id=e5eea92fbfc74ad2b915c81df03779a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:43.128767 14890 tablet_service.cc:1467] Processing CreateTablet for tablet b849e9505f5049ffa7fe153d355c3953 (DEFAULT_TABLE table=test-table [id=e5eea92fbfc74ad2b915c81df03779a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:43.128892 14597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b849e9505f5049ffa7fe153d355c3953. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:43.128923 14743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b849e9505f5049ffa7fe153d355c3953. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:43.129204 14890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b849e9505f5049ffa7fe153d355c3953. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:43.135855 15079 tablet_bootstrap.cc:492] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1: Bootstrap starting.
I20260809 20:23:43.137539 15078 tablet_bootstrap.cc:492] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95: Bootstrap starting.
I20260809 20:23:43.138243 15079 tablet_bootstrap.cc:654] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:43.139849 15078 tablet_bootstrap.cc:654] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:43.146140 15080 tablet_bootstrap.cc:492] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69: Bootstrap starting.
I20260809 20:23:43.146809 15079 tablet_bootstrap.cc:492] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1: No bootstrap required, opened a new log
I20260809 20:23:43.147003 15079 ts_tablet_manager.cc:1397] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20260809 20:23:43.147751 15079 raft_consensus.cc:348] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.147987 15079 raft_consensus.cc:374] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:43.148098 15079 raft_consensus.cc:729] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8264ce544740fbaafde6a9e59659d1, State: Initialized, Role: FOLLOWER
I20260809 20:23:43.148417 15079 consensus_queue.cc:260] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.149765 15079 ts_tablet_manager.cc:1428] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260809 20:23:43.150676 15080 tablet_bootstrap.cc:654] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:43.152801 15078 tablet_bootstrap.cc:492] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95: No bootstrap required, opened a new log
I20260809 20:23:43.153100 15078 ts_tablet_manager.cc:1397] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.006s
I20260809 20:23:43.153776 15080 tablet_bootstrap.cc:492] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69: No bootstrap required, opened a new log
I20260809 20:23:43.154021 15080 ts_tablet_manager.cc:1397] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260809 20:23:43.153924 15078 raft_consensus.cc:348] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.154340 15078 raft_consensus.cc:374] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:43.154558 15078 raft_consensus.cc:729] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc20402607e47b39946bf6014a57c95, State: Initialized, Role: FOLLOWER
I20260809 20:23:43.154872 15078 consensus_queue.cc:260] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.155124 15080 raft_consensus.cc:348] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.155380 15080 raft_consensus.cc:374] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:43.155545 15080 raft_consensus.cc:729] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2966bfabe5cf4693bf89c970c3426e69, State: Initialized, Role: FOLLOWER
I20260809 20:23:43.155875 15080 consensus_queue.cc:260] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.156091 15078 ts_tablet_manager.cc:1428] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 20:23:43.156934 15080 ts_tablet_manager.cc:1428] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 20:23:43.230572 15083 raft_consensus.cc:482] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:23:43.230885 15083 raft_consensus.cc:504] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.231822 15083 leader_election.cc:288] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2966bfabe5cf4693bf89c970c3426e69 (127.12.78.2:42703), ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787)
I20260809 20:23:43.232530 14769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b849e9505f5049ffa7fe153d355c3953" candidate_uuid: "ff8264ce544740fbaafde6a9e59659d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2966bfabe5cf4693bf89c970c3426e69" is_pre_election: true
I20260809 20:23:43.232647 14916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b849e9505f5049ffa7fe153d355c3953" candidate_uuid: "ff8264ce544740fbaafde6a9e59659d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc20402607e47b39946bf6014a57c95" is_pre_election: true
I20260809 20:23:43.232883 14916 raft_consensus.cc:2454] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff8264ce544740fbaafde6a9e59659d1 in term 0.
I20260809 20:23:43.232858 14769 raft_consensus.cc:2454] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff8264ce544740fbaafde6a9e59659d1 in term 0.
I20260809 20:23:43.233383 14546 leader_election.cc:302] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ebc20402607e47b39946bf6014a57c95, ff8264ce544740fbaafde6a9e59659d1; no voters:
I20260809 20:23:43.233692 15083 raft_consensus.cc:2781] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:23:43.233830 15083 raft_consensus.cc:482] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:23:43.233944 15083 raft_consensus.cc:3037] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:43.236217 15083 raft_consensus.cc:504] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.236913 15083 leader_election.cc:288] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [CANDIDATE]: Term 1 election: Requested vote from peers 2966bfabe5cf4693bf89c970c3426e69 (127.12.78.2:42703), ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787)
I20260809 20:23:43.237473 14769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b849e9505f5049ffa7fe153d355c3953" candidate_uuid: "ff8264ce544740fbaafde6a9e59659d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2966bfabe5cf4693bf89c970c3426e69"
I20260809 20:23:43.237512 14916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b849e9505f5049ffa7fe153d355c3953" candidate_uuid: "ff8264ce544740fbaafde6a9e59659d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc20402607e47b39946bf6014a57c95"
I20260809 20:23:43.237730 14769 raft_consensus.cc:3037] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:43.237778 14916 raft_consensus.cc:3037] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:43.240226 14769 raft_consensus.cc:2454] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff8264ce544740fbaafde6a9e59659d1 in term 1.
I20260809 20:23:43.240703 14548 leader_election.cc:302] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2966bfabe5cf4693bf89c970c3426e69, ff8264ce544740fbaafde6a9e59659d1; no voters:
I20260809 20:23:43.241014 14916 raft_consensus.cc:2454] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff8264ce544740fbaafde6a9e59659d1 in term 1.
I20260809 20:23:43.241080 15083 raft_consensus.cc:2781] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:43.241914 15083 raft_consensus.cc:686] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [term 1 LEADER]: Becoming Leader. State: Replica: ff8264ce544740fbaafde6a9e59659d1, State: Running, Role: LEADER
I20260809 20:23:43.242224 15083 consensus_queue.cc:237] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.245021 14454 catalog_manager.cc:5495] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff8264ce544740fbaafde6a9e59659d1 (127.12.78.1). New cstate: current_term: 1 leader_uuid: "ff8264ce544740fbaafde6a9e59659d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } health_report { overall_health: UNKNOWN } } }
I20260809 20:23:43.285935 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:43.280083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.12.78.1:35113)
0809 20:23:43.280672 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:43.280676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:43.280695 (+ 19us) server_negotiation.cc:407] Connection header received
0809 20:23:43.280748 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:43.280752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:43.280812 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:43.281036 (+ 224us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:23:43.281527 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:43.282386 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:43.284103 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:43.284520 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:43.284682 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:43.285088 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:23:43.285208 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:23:43.285277 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:43.285405 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 20:23:43.285468 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":317,"threads_started":1}
I20260809 20:23:43.296777 14769 raft_consensus.cc:1261] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Refusing update from remote peer ff8264ce544740fbaafde6a9e59659d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:43.296808 14916 raft_consensus.cc:1261] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Refusing update from remote peer ff8264ce544740fbaafde6a9e59659d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:43.297422 15086 consensus_queue.cc:1035] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:43.297753 15083 consensus_queue.cc:1035] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:43.347330 14454 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:23:43.347692 14454 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:23:43.351408 14454 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39: Sending DeleteTablet for 3 replicas of tablet b849e9505f5049ffa7fe153d355c3953
I20260809 20:23:43.352527 14743 tablet_service.cc:1514] Processing DeleteTablet for tablet b849e9505f5049ffa7fe153d355c3953 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:43 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:47122
I20260809 20:23:43.352625 14890 tablet_service.cc:1514] Processing DeleteTablet for tablet b849e9505f5049ffa7fe153d355c3953 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:43 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:57696
I20260809 20:23:43.353003 15068 tablet_replica.cc:330] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95: stopping tablet replica
I20260809 20:23:43.353261 15066 tablet_replica.cc:330] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69: stopping tablet replica
I20260809 20:23:43.353291 15068 raft_consensus.cc:2229] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:23:43.353550 15066 raft_consensus.cc:2229] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:23:43.353559 15068 raft_consensus.cc:2258] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:43.353777 15066 raft_consensus.cc:2258] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:43.353787 14455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 20:23:43.354341 14597 tablet_service.cc:1514] Processing DeleteTablet for tablet b849e9505f5049ffa7fe153d355c3953 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:43 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:45350
W20260809 20:23:43.354511 14455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:23:43.354580 15067 tablet_replica.cc:330] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1: stopping tablet replica
I20260809 20:23:43.354774 15067 raft_consensus.cc:2229] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:23:43.355114 15067 raft_consensus.cc:2258] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:43.355108 15068 ts_tablet_manager.cc:1905] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:43.356294 15066 ts_tablet_manager.cc:1905] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:43.357956 15067 ts_tablet_manager.cc:1905] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:43.362845 15068 ts_tablet_manager.cc:1918] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:23:43.363929 15068 log.cc:1196] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/b849e9505f5049ffa7fe153d355c3953
I20260809 20:23:43.364529 15068 ts_tablet_manager.cc:1939] T b849e9505f5049ffa7fe153d355c3953 P ebc20402607e47b39946bf6014a57c95: Deleting consensus metadata
I20260809 20:23:43.365345 15066 ts_tablet_manager.cc:1918] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:23:43.365501 15066 log.cc:1196] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/b849e9505f5049ffa7fe153d355c3953
I20260809 20:23:43.365906 15066 ts_tablet_manager.cc:1939] T b849e9505f5049ffa7fe153d355c3953 P 2966bfabe5cf4693bf89c970c3426e69: Deleting consensus metadata
I20260809 20:23:43.366447 14439 catalog_manager.cc:4841] TS ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787): tablet b849e9505f5049ffa7fe153d355c3953 (table test-table [id=e5eea92fbfc74ad2b915c81df03779a1]) successfully deleted
I20260809 20:23:43.366887 14441 catalog_manager.cc:4841] TS 2966bfabe5cf4693bf89c970c3426e69 (127.12.78.2:42703): tablet b849e9505f5049ffa7fe153d355c3953 (table test-table [id=e5eea92fbfc74ad2b915c81df03779a1]) successfully deleted
I20260809 20:23:43.369529 14597 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4c87d83b0b45b7b4677125ef0c577f (DEFAULT_TABLE table=test-table [id=0ba4f99bdb0b4f7f80db536fd003aab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:43.369820 15067 ts_tablet_manager.cc:1918] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:23:43.369963 15067 log.cc:1196] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/b849e9505f5049ffa7fe153d355c3953
I20260809 20:23:43.370064 14597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4c87d83b0b45b7b4677125ef0c577f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:43.370152 14890 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4c87d83b0b45b7b4677125ef0c577f (DEFAULT_TABLE table=test-table [id=0ba4f99bdb0b4f7f80db536fd003aab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:43.370139 14743 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4c87d83b0b45b7b4677125ef0c577f (DEFAULT_TABLE table=test-table [id=0ba4f99bdb0b4f7f80db536fd003aab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:23:43.370455 15067 ts_tablet_manager.cc:1939] T b849e9505f5049ffa7fe153d355c3953 P ff8264ce544740fbaafde6a9e59659d1: Deleting consensus metadata
I20260809 20:23:43.370546 14743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4c87d83b0b45b7b4677125ef0c577f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:43.370708 14890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4c87d83b0b45b7b4677125ef0c577f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:43.372623 14439 catalog_manager.cc:4841] TS ff8264ce544740fbaafde6a9e59659d1 (127.12.78.1:35113): tablet b849e9505f5049ffa7fe153d355c3953 (table test-table [id=e5eea92fbfc74ad2b915c81df03779a1]) successfully deleted
I20260809 20:23:43.376781 15079 tablet_bootstrap.cc:492] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1: Bootstrap starting.
I20260809 20:23:43.378521 15080 tablet_bootstrap.cc:492] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69: Bootstrap starting.
I20260809 20:23:43.379192 15079 tablet_bootstrap.cc:654] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:43.381126 15080 tablet_bootstrap.cc:654] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:43.381230 15078 tablet_bootstrap.cc:492] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95: Bootstrap starting.
I20260809 20:23:43.382427 15079 tablet_bootstrap.cc:492] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1: No bootstrap required, opened a new log
I20260809 20:23:43.382727 15079 ts_tablet_manager.cc:1397] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260809 20:23:43.383229 15078 tablet_bootstrap.cc:654] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:43.383761 15079 raft_consensus.cc:348] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.384239 15079 raft_consensus.cc:374] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:43.384384 15079 raft_consensus.cc:729] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8264ce544740fbaafde6a9e59659d1, State: Initialized, Role: FOLLOWER
I20260809 20:23:43.384887 15079 consensus_queue.cc:260] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.385704 15079 ts_tablet_manager.cc:1428] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 20:23:43.385978 15078 tablet_bootstrap.cc:492] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95: No bootstrap required, opened a new log
I20260809 20:23:43.386225 15078 ts_tablet_manager.cc:1397] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260809 20:23:43.386622 15080 tablet_bootstrap.cc:492] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69: No bootstrap required, opened a new log
I20260809 20:23:43.386945 15080 ts_tablet_manager.cc:1397] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 20:23:43.387420 15078 raft_consensus.cc:348] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.387840 15078 raft_consensus.cc:374] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:43.387989 15078 raft_consensus.cc:729] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc20402607e47b39946bf6014a57c95, State: Initialized, Role: FOLLOWER
I20260809 20:23:43.388200 15080 raft_consensus.cc:348] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.388612 15080 raft_consensus.cc:374] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:43.388413 15078 consensus_queue.cc:260] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.388751 15080 raft_consensus.cc:729] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2966bfabe5cf4693bf89c970c3426e69, State: Initialized, Role: FOLLOWER
I20260809 20:23:43.389088 15080 consensus_queue.cc:260] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.389700 15080 ts_tablet_manager.cc:1428] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 20:23:43.390234 15078 ts_tablet_manager.cc:1428] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260809 20:23:43.423306 15011 raft_consensus.cc:482] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:23:43.423646 15011 raft_consensus.cc:504] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.424597 15011 leader_election.cc:288] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8264ce544740fbaafde6a9e59659d1 (127.12.78.1:35113), ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787)
I20260809 20:23:43.425149 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4c87d83b0b45b7b4677125ef0c577f" candidate_uuid: "2966bfabe5cf4693bf89c970c3426e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8264ce544740fbaafde6a9e59659d1" is_pre_election: true
I20260809 20:23:43.425544 14619 raft_consensus.cc:2454] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2966bfabe5cf4693bf89c970c3426e69 in term 0.
I20260809 20:23:43.425479 14917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4c87d83b0b45b7b4677125ef0c577f" candidate_uuid: "2966bfabe5cf4693bf89c970c3426e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc20402607e47b39946bf6014a57c95" is_pre_election: true
I20260809 20:23:43.425805 14917 raft_consensus.cc:2454] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2966bfabe5cf4693bf89c970c3426e69 in term 0.
I20260809 20:23:43.426209 14696 leader_election.cc:302] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2966bfabe5cf4693bf89c970c3426e69, ff8264ce544740fbaafde6a9e59659d1; no voters:
I20260809 20:23:43.426630 15011 raft_consensus.cc:2781] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:23:43.426800 15011 raft_consensus.cc:482] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:23:43.427037 15011 raft_consensus.cc:3037] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:43.429661 15011 raft_consensus.cc:504] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.430321 15011 leader_election.cc:288] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [CANDIDATE]: Term 1 election: Requested vote from peers ff8264ce544740fbaafde6a9e59659d1 (127.12.78.1:35113), ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787)
I20260809 20:23:43.430902 14917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4c87d83b0b45b7b4677125ef0c577f" candidate_uuid: "2966bfabe5cf4693bf89c970c3426e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc20402607e47b39946bf6014a57c95"
I20260809 20:23:43.430887 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4c87d83b0b45b7b4677125ef0c577f" candidate_uuid: "2966bfabe5cf4693bf89c970c3426e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8264ce544740fbaafde6a9e59659d1"
I20260809 20:23:43.431244 14917 raft_consensus.cc:3037] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:43.431263 14619 raft_consensus.cc:3037] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:43.435024 14619 raft_consensus.cc:2454] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2966bfabe5cf4693bf89c970c3426e69 in term 1.
I20260809 20:23:43.435043 14917 raft_consensus.cc:2454] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2966bfabe5cf4693bf89c970c3426e69 in term 1.
I20260809 20:23:43.435443 14696 leader_election.cc:302] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2966bfabe5cf4693bf89c970c3426e69, ebc20402607e47b39946bf6014a57c95; no voters:
I20260809 20:23:43.435730 15011 raft_consensus.cc:2781] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:43.435981 15011 raft_consensus.cc:686] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [term 1 LEADER]: Becoming Leader. State: Replica: 2966bfabe5cf4693bf89c970c3426e69, State: Running, Role: LEADER
I20260809 20:23:43.436296 15011 consensus_queue.cc:237] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } }
I20260809 20:23:43.439613 14455 catalog_manager.cc:5495] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2966bfabe5cf4693bf89c970c3426e69 (127.12.78.2). New cstate: current_term: 1 leader_uuid: "2966bfabe5cf4693bf89c970c3426e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2966bfabe5cf4693bf89c970c3426e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 } health_report { overall_health: UNKNOWN } } }
I20260809 20:23:43.474931 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:43.468608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.12.78.2:42703)
0809 20:23:43.469385 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:43.469391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:43.469415 (+ 24us) server_negotiation.cc:407] Connection header received
0809 20:23:43.469527 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:43.469531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:43.469593 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:43.469733 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:23:43.470249 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:43.471298 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:43.472897 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:43.473391 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:43.473540 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:43.474042 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:23:43.474160 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:23:43.474268 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:43.474432 (+ 164us) server_negotiation.cc:299] Negotiation successful
0809 20:23:43.474511 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":494,"threads_started":1}
I20260809 20:23:43.484874 14917 raft_consensus.cc:1261] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Refusing update from remote peer 2966bfabe5cf4693bf89c970c3426e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:43.484961 14619 raft_consensus.cc:1261] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Refusing update from remote peer 2966bfabe5cf4693bf89c970c3426e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:23:43.485551 15055 consensus_queue.cc:1035] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebc20402607e47b39946bf6014a57c95" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:23:43.486025 15049 consensus_queue.cc:1035] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8264ce544740fbaafde6a9e59659d1" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 20:23:43.508963 15047 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 20:23:45.096778 14455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:23:45.097090 14455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:23:45.101549 14455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 950311ec2bfe4fc983d4596f79fcfe39: Sending DeleteTablet for 3 replicas of tablet 8d4c87d83b0b45b7b4677125ef0c577f
I20260809 20:23:45.102490 14597 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4c87d83b0b45b7b4677125ef0c577f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:45 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:45350
I20260809 20:23:45.102523 14743 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4c87d83b0b45b7b4677125ef0c577f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:45 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:47122
I20260809 20:23:45.103269 14890 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4c87d83b0b45b7b4677125ef0c577f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:23:45 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:57696
I20260809 20:23:45.103332 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 14527
I20260809 20:23:45.103332 15121 tablet_replica.cc:330] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69: stopping tablet replica
I20260809 20:23:45.103616 15121 raft_consensus.cc:2229] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:23:45.104292 15121 raft_consensus.cc:2258] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:45.105679 15122 tablet_replica.cc:330] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1: stopping tablet replica
I20260809 20:23:45.105849 15121 ts_tablet_manager.cc:1905] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:45.105926 15122 raft_consensus.cc:2229] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:23:45.106185 15122 raft_consensus.cc:2258] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:45.108729 15122 ts_tablet_manager.cc:1905] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:45.113300 15121 ts_tablet_manager.cc:1918] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 20:23:45.113548 15121 log.cc:1196] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/8d4c87d83b0b45b7b4677125ef0c577f
I20260809 20:23:45.114220 15121 ts_tablet_manager.cc:1939] T 8d4c87d83b0b45b7b4677125ef0c577f P 2966bfabe5cf4693bf89c970c3426e69: Deleting consensus metadata
I20260809 20:23:45.115535 14441 catalog_manager.cc:4841] TS 2966bfabe5cf4693bf89c970c3426e69 (127.12.78.2:42703): tablet 8d4c87d83b0b45b7b4677125ef0c577f (table test-table [id=0ba4f99bdb0b4f7f80db536fd003aab7]) successfully deleted
I20260809 20:23:45.116854 15122 ts_tablet_manager.cc:1918] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 20:23:45.117147 15123 tablet_replica.cc:330] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95: stopping tablet replica
I20260809 20:23:45.117383 15122 log.cc:1196] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/8d4c87d83b0b45b7b4677125ef0c577f
I20260809 20:23:45.117488 15123 raft_consensus.cc:2229] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:23:45.117839 15123 raft_consensus.cc:2258] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:23:45.118180 15122 ts_tablet_manager.cc:1939] T 8d4c87d83b0b45b7b4677125ef0c577f P ff8264ce544740fbaafde6a9e59659d1: Deleting consensus metadata
I20260809 20:23:45.119238 14439 catalog_manager.cc:4841] TS ff8264ce544740fbaafde6a9e59659d1 (127.12.78.1:35113): tablet 8d4c87d83b0b45b7b4677125ef0c577f (table test-table [id=0ba4f99bdb0b4f7f80db536fd003aab7]) successfully deleted
I20260809 20:23:45.119257 15123 ts_tablet_manager.cc:1905] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:23:45.125113 15123 ts_tablet_manager.cc:1918] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 20:23:45.125232 15123 log.cc:1196] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/8d4c87d83b0b45b7b4677125ef0c577f
I20260809 20:23:45.125572 15123 ts_tablet_manager.cc:1939] T 8d4c87d83b0b45b7b4677125ef0c577f P ebc20402607e47b39946bf6014a57c95: Deleting consensus metadata
I20260809 20:23:45.126590 14439 catalog_manager.cc:4841] TS ebc20402607e47b39946bf6014a57c95 (127.12.78.3:34787): tablet 8d4c87d83b0b45b7b4677125ef0c577f (table test-table [id=0ba4f99bdb0b4f7f80db536fd003aab7]) successfully deleted
Aug 09 20:23:45 dist-test-slave-rnrw krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:23:45.136420 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:45.106513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.12.78.1:35113)
0809 20:23:45.117630 (+ 11117us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:45.117637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:45.117668 (+ 31us) server_negotiation.cc:407] Connection header received
0809 20:23:45.117768 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:45.117772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:45.117839 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:45.117986 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:45.119772 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:45.120512 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:45.122352 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:45.122769 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:45.122864 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:45.130759 (+ 7895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:45.130777 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:45.130793 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:45.130831 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:45.133923 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:45.134617 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:45.134624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:45.134626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:45.134766 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:45.135105 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:45.135108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:45.135110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:45.135478 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:45.135694 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:45.135957 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 20:23:45.136138 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10915,"thread_start_us":10551,"threads_started":1}
W20260809 20:23:45.137251 14667 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45384
W20260809 20:23:45.137902 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 14527: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:23:45.138154 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 14527
I20260809 20:23:45.165602 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 14677
Aug 09 20:23:45 dist-test-slave-rnrw krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:23:45.182310 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:45.166509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.12.78.2:42703)
0809 20:23:45.167079 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:45.167083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:45.167099 (+ 16us) server_negotiation.cc:407] Connection header received
0809 20:23:45.167157 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:45.167160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:45.167221 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:45.167357 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:45.168770 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:45.169769 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:45.171079 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:45.171493 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:45.171635 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:45.176183 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:45.176202 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:45.176219 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:45.176259 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:45.179302 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:45.180044 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:45.180052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:45.180055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:45.180200 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:45.180631 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:45.180643 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:45.180646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:45.181035 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:45.181359 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:45.181687 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 20:23:45.181935 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":320,"threads_started":1}
W20260809 20:23:45.183084 14810 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47150
W20260809 20:23:45.183743 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 14677: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:23:45.183884 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 14677
I20260809 20:23:45.213404 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 14819
Aug 09 20:23:45 dist-test-slave-rnrw krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:23:45.229970 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:45.214252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.12.78.3:34787)
0809 20:23:45.214811 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:45.214816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:45.214833 (+ 17us) server_negotiation.cc:407] Connection header received
0809 20:23:45.214890 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:45.214894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:45.214956 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:45.215073 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:45.216413 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:45.217285 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:45.218589 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:45.218992 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:45.219089 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:45.223521 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:45.223538 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:45.223553 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:45.223595 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:45.226806 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:45.227540 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:45.227549 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:45.227551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:45.227714 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:45.228061 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:45.228065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:45.228067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:45.228431 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:45.228866 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:45.229123 (+ 257us) server_negotiation.cc:299] Negotiation successful
0809 20:23:45.229370 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":314,"threads_started":1}
W20260809 20:23:45.230967 14959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57698
W20260809 20:23:45.231719 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 14819: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:23:45.231926 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 14819
I20260809 20:23:45.255506 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 14421
I20260809 20:23:45.256894 14496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:45.374637 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 14421
2026-08-09T20:23:45Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8482 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:23:45 dist-test-slave-rnrw krb5kdc[15131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:23:45 dist-test-slave-rnrw krb5kdc[15131](info): set up 2 sockets
Aug 09 20:23:45 dist-test-slave-rnrw krb5kdc[15131](info): commencing operation
krb5kdc: starting...
W20260809 20:23:46.586469 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260809 20:23:46.714913 12600 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.128s user 0.001s sys 0.006s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260809 20:23:46.880316 12600 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.165s user 0.000s sys 0.006s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260809 20:23:47.109547 12600 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.229s user 0.000s sys 0.006s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/test-user.keytab.
W20260809 20:23:47.583345 12600 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.474s user 0.000s sys 0.006s
Aug 09 20:23:47 dist-test-slave-rnrw krb5kdc[15131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:23:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:23:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
W20260809 20:23:47.899593 12600 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.12.78.62: real 0.254s user 0.002s sys 0.011s
I20260809 20:23:47.900784 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:33387
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:33387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:48.246832 15145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:48.247179 15145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:48.247273 15145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:48.254838 15145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:23:48.254961 15145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:48.255007 15145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:48.255059 15145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:23:48.255095 15145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:23:48.265177 15145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:33387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:33387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:48.266536 15145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:48.267915 15145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:48.280630 15153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:48.280673 15151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:48.280577 15150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:48.281394 15145 server_base.cc:1034] running on GCE node
I20260809 20:23:48.282212 15145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:48.283702 15145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:48.284933 15145 hybrid_clock.cc:648] HybridClock initialized: now 1786307028284884 us; error 57 us; skew 500 ppm
I20260809 20:23:48.285288 15145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:48 dist-test-slave-rnrw krb5kdc[15131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307028, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:48.289129 15145 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:23:48.290691 15145 webserver.cc:460] Webserver started at http://127.12.78.62:32973/ using document root <none> and password file <none>
I20260809 20:23:48.291437 15145 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:48.291601 15145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:48.291939 15145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:48.295142 15145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "e6b2d63e007842478d6296e1a0b69230"
format_stamp: "Formatted at 2026-08-09 20:23:48 on dist-test-slave-rnrw"
server_key: "0171f113c4e895f9f22f685dab732b41"
server_key_iv: "3d61b8f63af2a2a72fd233e69581ad03"
server_key_version: "encryptionkey@0"
I20260809 20:23:48.295964 15145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "e6b2d63e007842478d6296e1a0b69230"
format_stamp: "Formatted at 2026-08-09 20:23:48 on dist-test-slave-rnrw"
server_key: "0171f113c4e895f9f22f685dab732b41"
server_key_iv: "3d61b8f63af2a2a72fd233e69581ad03"
server_key_version: "encryptionkey@0"
I20260809 20:23:48.301556 15145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260809 20:23:48.305079 15161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:48.306486 15145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 20:23:48.306764 15145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "e6b2d63e007842478d6296e1a0b69230"
format_stamp: "Formatted at 2026-08-09 20:23:48 on dist-test-slave-rnrw"
server_key: "0171f113c4e895f9f22f685dab732b41"
server_key_iv: "3d61b8f63af2a2a72fd233e69581ad03"
server_key_version: "encryptionkey@0"
I20260809 20:23:48.306977 15145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:48.322420 15145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:48.326534 15145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:48.326834 15145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:48.346223 15145 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:33387
I20260809 20:23:48.346293 15214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:33387 every 8 connection(s)
I20260809 20:23:48.347780 15145 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:23:48.351707 15215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:48.353204 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 15145
I20260809 20:23:48.353426 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:23:48.354094 12600 external_mini_cluster.cc:1428] Setting key 2b5bdb39eec2bfd3d80542778159016b
I20260809 20:23:48.366585 15215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230: Bootstrap starting.
Aug 09 20:23:48 dist-test-slave-rnrw krb5kdc[15131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:48.370216 15215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:48.371500 15215 log.cc:824] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:48.374783 15215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230: No bootstrap required, opened a new log
I20260809 20:23:48.378400 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:48.356156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.12.78.62:33387)
0809 20:23:48.357082 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:48.357103 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:48.357154 (+ 51us) server_negotiation.cc:407] Connection header received
0809 20:23:48.358890 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:48.358920 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:48.359464 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:48.359906 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:48.361308 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:48.363399 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:48.364735 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:48.365761 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:48.365942 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:48.370498 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:48.370506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:48.370520 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:48.370566 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:48.374120 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:48.374999 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:48.375005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:48.375017 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:48.375190 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:48.375608 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:48.375611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:48.375613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:48.376133 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:48.376343 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:48.376956 (+ 613us) server_negotiation.cc:299] Negotiation successful
0809 20:23:48.377375 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":402,"threads_started":1}
I20260809 20:23:48.383651 15215 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b2d63e007842478d6296e1a0b69230" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33387 } }
I20260809 20:23:48.384069 15215 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:48.384202 15215 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b2d63e007842478d6296e1a0b69230, State: Initialized, Role: FOLLOWER
I20260809 20:23:48.384832 15215 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b2d63e007842478d6296e1a0b69230" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33387 } }
I20260809 20:23:48.385075 15215 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:23:48.385208 15215 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:23:48.385396 15215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:48.387562 15215 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b2d63e007842478d6296e1a0b69230" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33387 } }
I20260809 20:23:48.388098 15215 leader_election.cc:302] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6b2d63e007842478d6296e1a0b69230; no voters:
I20260809 20:23:48.388748 15215 leader_election.cc:288] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:23:48.388890 15220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:48.389559 15220 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [term 1 LEADER]: Becoming Leader. State: Replica: e6b2d63e007842478d6296e1a0b69230, State: Running, Role: LEADER
I20260809 20:23:48.390121 15220 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b2d63e007842478d6296e1a0b69230" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33387 } }
I20260809 20:23:48.390843 15215 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:23:48.392886 15221 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6b2d63e007842478d6296e1a0b69230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b2d63e007842478d6296e1a0b69230" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33387 } } }
I20260809 20:23:48.393276 15221 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:48.393996 15222 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6b2d63e007842478d6296e1a0b69230. Latest consensus state: current_term: 1 leader_uuid: "e6b2d63e007842478d6296e1a0b69230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b2d63e007842478d6296e1a0b69230" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33387 } } }
I20260809 20:23:48.394264 15227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:23:48.395171 15222 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230 [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:48.400503 15227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:23:48.410051 15227 catalog_manager.cc:1347] Generated new cluster ID: 8cb987c03bce489991c2a3355ed5edf2
I20260809 20:23:48.410261 15227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:23:48.421196 15227 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:23:48.422609 15227 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:23:48.428884 15227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6b2d63e007842478d6296e1a0b69230: Generated new TSK 0
I20260809 20:23:48.430068 15227 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:23:48.488379 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33387
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:48.831735 15241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:48.832070 15241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:48.832240 15241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:48.839818 15241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:48.839967 15241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:48.840159 15241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:23:48.850311 15241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:48.851662 15241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:48.853099 15241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:48.866393 15246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:48.866339 15249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:48.866305 15247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:48.867370 15241 server_base.cc:1034] running on GCE node
I20260809 20:23:48.868031 15241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:48.869036 15241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:48.870237 15241 hybrid_clock.cc:648] HybridClock initialized: now 1786307028870175 us; error 52 us; skew 500 ppm
I20260809 20:23:48.870558 15241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:48 dist-test-slave-rnrw krb5kdc[15131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307028, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:48.873821 15241 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:23:48.875303 15241 webserver.cc:460] Webserver started at http://127.12.78.1:37625/ using document root <none> and password file <none>
I20260809 20:23:48.876017 15241 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:48.876151 15241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:48.876471 15241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:48.879524 15241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "655d85773d2f47f9a369c7db68b69e1f"
format_stamp: "Formatted at 2026-08-09 20:23:48 on dist-test-slave-rnrw"
server_key: "33c3f9af8189c8b5b14bb180a25c249e"
server_key_iv: "1e47ade58617571fb0c579c83eb485d1"
server_key_version: "encryptionkey@0"
I20260809 20:23:48.880434 15241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "655d85773d2f47f9a369c7db68b69e1f"
format_stamp: "Formatted at 2026-08-09 20:23:48 on dist-test-slave-rnrw"
server_key: "33c3f9af8189c8b5b14bb180a25c249e"
server_key_iv: "1e47ade58617571fb0c579c83eb485d1"
server_key_version: "encryptionkey@0"
I20260809 20:23:48.885803 15241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 20:23:48.889321 15257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:48.890887 15241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:23:48.891113 15241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "655d85773d2f47f9a369c7db68b69e1f"
format_stamp: "Formatted at 2026-08-09 20:23:48 on dist-test-slave-rnrw"
server_key: "33c3f9af8189c8b5b14bb180a25c249e"
server_key_iv: "1e47ade58617571fb0c579c83eb485d1"
server_key_version: "encryptionkey@0"
I20260809 20:23:48.891335 15241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:48.917870 15241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:48.922336 15241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:48.922662 15241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:48.923705 15241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:48.925201 15241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:48.925315 15241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:48.925477 15241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:48.925550 15241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:48.959441 15241 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:39519
I20260809 20:23:48.959488 15375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:39519 every 8 connection(s)
I20260809 20:23:48.960866 15241 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:23:48.961475 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 15241
I20260809 20:23:48.961700 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:23:48.962335 12600 external_mini_cluster.cc:1428] Setting key 19e9d385aba3e29f9b619baa88760eb4
Aug 09 20:23:48 dist-test-slave-rnrw krb5kdc[15131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307028, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:48.984498 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:48.964114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:41403 (local address 127.12.78.62:33387)
0809 20:23:48.964517 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:48.964521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:48.965872 (+ 1351us) server_negotiation.cc:407] Connection header received
0809 20:23:48.967774 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:48.967780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:48.967876 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:48.968000 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:48.969720 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:48.970444 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:48.972005 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:48.972773 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:48.972911 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:48.977214 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:48.977223 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:48.977231 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:48.977282 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:48.980985 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:48.981782 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:48.981790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:48.981795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:48.981964 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:48.982438 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:48.982446 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:48.982453 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:48.982886 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:48.983070 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:48.983803 (+ 733us) server_negotiation.cc:299] Negotiation successful
0809 20:23:48.984065 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":209,"threads_started":1}
I20260809 20:23:48.985293 15379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:48.964867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:33387 (local address 127.12.78.1:41403)
0809 20:23:48.965717 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:48.965757 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:48.967499 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:48.968281 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:48.968290 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:48.968940 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:48.969534 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:48.969552 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:48.970623 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:48.970627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:48.971849 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:48.971867 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:48.973023 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:48.973029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:48.973236 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:48.973972 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:48.974006 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:48.976961 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:48.981141 (+ 4180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:48.981149 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:48.981164 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:48.981625 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:48.982096 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:48.982100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:48.982103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:48.982305 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:48.983017 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:48.983024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:48.983375 (+ 351us) client_negotiation.cc:769] Sending connection context
0809 20:23:48.983609 (+ 234us) client_negotiation.cc:240] Negotiation successful
0809 20:23:48.984043 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":328,"threads_started":1}
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
I20260809 20:23:48.987066 15376 heartbeater.cc:347] Connected to a master server at 127.12.78.62:33387
I20260809 20:23:48.987425 15376 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:48.988227 15376 heartbeater.cc:511] Master 127.12.78.62:33387 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
I20260809 20:23:48.990545 15179 ts_manager.cc:194] Registered new tserver with Master: 655d85773d2f47f9a369c7db68b69e1f (127.12.78.1:39519)
I20260809 20:23:48.992266 15179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:41403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:23:49.016665 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33387
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:49.358553 15382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:49.358906 15382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:49.359095 15382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:49.366916 15382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:49.367096 15382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:49.367290 15382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:23:49.378105 15382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:49.379520 15382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:49.380959 15382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:49.394313 15387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:49.394412 15388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:49.394347 15390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:49.398061 15382 server_base.cc:1034] running on GCE node
I20260809 20:23:49.398739 15382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:49.399730 15382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:49.400967 15382 hybrid_clock.cc:648] HybridClock initialized: now 1786307029400916 us; error 41 us; skew 500 ppm
I20260809 20:23:49.401288 15382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:49 dist-test-slave-rnrw krb5kdc[15131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307029, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:49.404388 15382 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:23:49.405877 15382 webserver.cc:460] Webserver started at http://127.12.78.2:42187/ using document root <none> and password file <none>
I20260809 20:23:49.406606 15382 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:49.406754 15382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:49.407073 15382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:49.410077 15382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "1d4712458cdd40de8cbc9d336f8ef6d1"
format_stamp: "Formatted at 2026-08-09 20:23:49 on dist-test-slave-rnrw"
server_key: "abce90200c8e115090d1dda123e79740"
server_key_iv: "1f314dbf11fe83b29d082563dacda196"
server_key_version: "encryptionkey@0"
I20260809 20:23:49.411017 15382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "1d4712458cdd40de8cbc9d336f8ef6d1"
format_stamp: "Formatted at 2026-08-09 20:23:49 on dist-test-slave-rnrw"
server_key: "abce90200c8e115090d1dda123e79740"
server_key_iv: "1f314dbf11fe83b29d082563dacda196"
server_key_version: "encryptionkey@0"
I20260809 20:23:49.416445 15382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 20:23:49.420166 15399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:49.421653 15382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 20:23:49.421850 15382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "1d4712458cdd40de8cbc9d336f8ef6d1"
format_stamp: "Formatted at 2026-08-09 20:23:49 on dist-test-slave-rnrw"
server_key: "abce90200c8e115090d1dda123e79740"
server_key_iv: "1f314dbf11fe83b29d082563dacda196"
server_key_version: "encryptionkey@0"
I20260809 20:23:49.422084 15382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:49.445835 15382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:49.450307 15382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:49.450727 15382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:49.451756 15382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:49.453166 15382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:49.453284 15382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:49.453444 15382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:49.453524 15382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:49.488185 15382 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:35377
I20260809 20:23:49.488231 15521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:35377 every 8 connection(s)
I20260809 20:23:49.489630 15382 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:23:49.498849 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 15382
I20260809 20:23:49.499053 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:23:49.499526 12600 external_mini_cluster.cc:1428] Setting key 81e4ba0a26a43b7abafbf78b09cdbd6a
Aug 09 20:23:49 dist-test-slave-rnrw krb5kdc[15131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307029, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:49.512989 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:49.492410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:55117 (local address 127.12.78.62:33387)
0809 20:23:49.492882 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:49.492887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:49.493963 (+ 1076us) server_negotiation.cc:407] Connection header received
0809 20:23:49.495972 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:49.495979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:49.496078 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:49.496215 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:49.498125 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:49.498883 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:49.500656 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:49.501157 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:49.501322 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:49.505118 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:49.505128 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:49.505137 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:49.505192 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:49.509263 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:49.510299 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:49.510303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:49.510306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:49.510502 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:49.511048 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:49.511051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:49.511054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:49.511355 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:49.511490 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:49.512416 (+ 926us) server_negotiation.cc:299] Negotiation successful
0809 20:23:49.512684 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":256,"threads_started":1}
I20260809 20:23:49.514503 15526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:49.493063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:33387 (local address 127.12.78.2:55117)
0809 20:23:49.493798 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:49.493836 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:49.495609 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:49.496533 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:49.496600 (+ 67us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:49.497270 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:49.497953 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:49.497973 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:49.499061 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:49.499065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:49.500441 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:49.500453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:49.501457 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:49.501465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:49.501753 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:49.502492 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:49.502516 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:49.504819 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:49.509425 (+ 4606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:49.509438 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:49.509459 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:49.510150 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:49.510637 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:49.510643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:49.510666 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:49.510924 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:49.511486 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:49.511498 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:49.512001 (+ 503us) client_negotiation.cc:769] Sending connection context
0809 20:23:49.512313 (+ 312us) client_negotiation.cc:240] Negotiation successful
0809 20:23:49.512970 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":287,"threads_started":1}
I20260809 20:23:49.516135 15522 heartbeater.cc:347] Connected to a master server at 127.12.78.62:33387
I20260809 20:23:49.516470 15522 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:49.517243 15522 heartbeater.cc:511] Master 127.12.78.62:33387 requested a full tablet report, sending...
I20260809 20:23:49.518817 15179 ts_manager.cc:194] Registered new tserver with Master: 1d4712458cdd40de8cbc9d336f8ef6d1 (127.12.78.2:35377)
I20260809 20:23:49.519639 15179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:55117
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:23:49.552659 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33387
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:49.887399 15529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:49.887724 15529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:49.887899 15529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:49.895252 15529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:49.895393 15529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:49.895597 15529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:23:49.905961 15529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:49.907326 15529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:49.908779 15529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:49.922726 15534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:49.922652 15537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:49.922695 15535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:49.923800 15529 server_base.cc:1034] running on GCE node
I20260809 20:23:49.924424 15529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:49.925452 15529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:49.926688 15529 hybrid_clock.cc:648] HybridClock initialized: now 1786307029926611 us; error 69 us; skew 500 ppm
I20260809 20:23:49.927057 15529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:49 dist-test-slave-rnrw krb5kdc[15131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307029, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:49.930327 15529 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:23:49.931829 15529 webserver.cc:460] Webserver started at http://127.12.78.3:37507/ using document root <none> and password file <none>
I20260809 20:23:49.932474 15529 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:49.932641 15529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:49.932988 15529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:49.935920 15529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "e08ff85c7df2456384eb601d0d6f9439"
format_stamp: "Formatted at 2026-08-09 20:23:49 on dist-test-slave-rnrw"
server_key: "263f62642f8557dd6ab04ecf0f02c2ba"
server_key_iv: "5b8f8b6c21d52cabdadb06f4ee852ccd"
server_key_version: "encryptionkey@0"
I20260809 20:23:49.936779 15529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "e08ff85c7df2456384eb601d0d6f9439"
format_stamp: "Formatted at 2026-08-09 20:23:49 on dist-test-slave-rnrw"
server_key: "263f62642f8557dd6ab04ecf0f02c2ba"
server_key_iv: "5b8f8b6c21d52cabdadb06f4ee852ccd"
server_key_version: "encryptionkey@0"
I20260809 20:23:49.941896 15529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 20:23:49.945194 15546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:49.946457 15529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:23:49.946628 15529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "e08ff85c7df2456384eb601d0d6f9439"
format_stamp: "Formatted at 2026-08-09 20:23:49 on dist-test-slave-rnrw"
server_key: "263f62642f8557dd6ab04ecf0f02c2ba"
server_key_iv: "5b8f8b6c21d52cabdadb06f4ee852ccd"
server_key_version: "encryptionkey@0"
I20260809 20:23:49.946834 15529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:49.959722 15529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:49.963966 15529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:49.964274 15529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:49.965253 15529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:49.966537 15529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:49.966611 15529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:49.966784 15529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:49.966838 15529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:49.996508 15376 heartbeater.cc:503] Master 127.12.78.62:33387 was elected leader, sending a full tablet report...
I20260809 20:23:50.001391 15529 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:38397
I20260809 20:23:50.001431 15664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:38397 every 8 connection(s)
I20260809 20:23:50.002712 15529 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:23:50.003826 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 15529
I20260809 20:23:50.004010 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:23:50.004413 12600 external_mini_cluster.cc:1428] Setting key 0c15484e05af7df7409a64e52528e890
Aug 09 20:23:50 dist-test-slave-rnrw krb5kdc[15131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307029, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:50.024119 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:50.005647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:51885 (local address 127.12.78.62:33387)
0809 20:23:50.005839 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:50.005844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:50.007127 (+ 1283us) server_negotiation.cc:407] Connection header received
0809 20:23:50.009261 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:50.009267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:50.009366 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:50.009483 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:50.011273 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:50.012069 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:50.013768 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:50.014175 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:50.014329 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:50.017843 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:50.017848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:50.017852 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:50.017885 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:50.020935 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:50.021720 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:50.021724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:50.021727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:50.021854 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:50.022320 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:50.022324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:50.022326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:50.022622 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:50.022807 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:50.023519 (+ 712us) server_negotiation.cc:299] Negotiation successful
0809 20:23:50.023775 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 20:23:50.025266 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:50.006093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:33387 (local address 127.12.78.3:51885)
0809 20:23:50.006968 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:50.007015 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:50.008971 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:50.009735 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:50.009744 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:50.010382 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:50.011078 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:50.011105 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:50.012229 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:50.012235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:50.013602 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:50.013615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:50.014431 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:50.014439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:50.014672 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:50.015309 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:50.015332 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:50.017634 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:50.021097 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:50.021118 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:50.021131 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:50.021568 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:50.021971 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:50.021976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:50.021987 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:50.022182 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:50.022766 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:50.022775 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:50.023136 (+ 361us) client_negotiation.cc:769] Sending connection context
0809 20:23:50.023413 (+ 277us) client_negotiation.cc:240] Negotiation successful
0809 20:23:50.023984 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"spinlock_wait_cycles":33536,"thread_start_us":408,"threads_started":1}
I20260809 20:23:50.026818 15665 heartbeater.cc:347] Connected to a master server at 127.12.78.62:33387
I20260809 20:23:50.027091 15665 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:50.027825 15665 heartbeater.cc:511] Master 127.12.78.62:33387 requested a full tablet report, sending...
I20260809 20:23:50.029224 15179 ts_manager.cc:194] Registered new tserver with Master: e08ff85c7df2456384eb601d0d6f9439 (127.12.78.3:38397)
I20260809 20:23:50.030012 15179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:51885
I20260809 20:23:50.033226 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 20:23:50.067188 15525 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:23:50.064754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.12.78.62:33387)
0809 20:23:50.065001 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:50.065007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:50.065227 (+ 220us) server_negotiation.cc:407] Connection header received
0809 20:23:50.065388 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:50.065392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:50.065470 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:50.065558 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:50.066880 (+ 1322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46828: BlockingRecv error: recv got EOF from 127.0.0.1:46828 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 20:23:50.070367 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 15241
W20260809 20:23:50.073254 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 15241: Not authorized: Client connection negotiation failed: client connection to 127.12.78.1:39519: server requires authentication, but client does not have Kerberos credentials available
I20260809 20:23:50.073477 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 15241
I20260809 20:23:50.091167 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 15382
W20260809 20:23:50.094031 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 15382: Not authorized: Client connection negotiation failed: client connection to 127.12.78.2:35377: server requires authentication, but client does not have Kerberos credentials available
I20260809 20:23:50.094203 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 15382
I20260809 20:23:50.113022 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 15529
W20260809 20:23:50.116128 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 15529: Not authorized: Client connection negotiation failed: client connection to 127.12.78.3:38397: server requires authentication, but client does not have Kerberos credentials available
I20260809 20:23:50.116314 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 15529
I20260809 20:23:50.133121 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 15145
I20260809 20:23:50.134476 15209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:50.239245 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 15145
2026-08-09T20:23:50Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4862 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:23:50 dist-test-slave-rnrw krb5kdc[15683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:23:50 dist-test-slave-rnrw krb5kdc[15683](info): set up 2 sockets
Aug 09 20:23:50 dist-test-slave-rnrw krb5kdc[15683](info): commencing operation
krb5kdc: starting...
W20260809 20:23:51.380273 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:23:51 dist-test-slave-rnrw krb5kdc[15683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:23:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:23:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:23:51.600759 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:45167
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:42579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:45167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:51.931633 15696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:51.931911 15696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:51.932085 15696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:51.939198 15696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:23:51.939339 15696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:51.939446 15696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:51.939558 15696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:23:51.939651 15696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:23:51.949589 15696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:42579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:45167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:45167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:51.950889 15696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:51.952214 15696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:51.963815 15702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:51.963845 15703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:51.963882 15705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:51.965103 15696 server_base.cc:1034] running on GCE node
I20260809 20:23:51.965940 15696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:51.967367 15696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:51.968655 15696 hybrid_clock.cc:648] HybridClock initialized: now 1786307031968598 us; error 51 us; skew 500 ppm
I20260809 20:23:51.969034 15696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:51 dist-test-slave-rnrw krb5kdc[15683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307031, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:51.972198 15696 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:23:51.973744 15696 webserver.cc:460] Webserver started at http://127.12.78.62:34821/ using document root <none> and password file <none>
I20260809 20:23:51.974443 15696 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:51.974589 15696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:51.974927 15696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:51.978039 15696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "cf6d4316de894d77b0d167c9812a03eb"
format_stamp: "Formatted at 2026-08-09 20:23:51 on dist-test-slave-rnrw"
server_key: "267b76bcff46e711dc374237b7c8af13"
server_key_iv: "9e07e50a6757ab4cac857c8907f0e2cb"
server_key_version: "encryptionkey@0"
I20260809 20:23:51.978871 15696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "cf6d4316de894d77b0d167c9812a03eb"
format_stamp: "Formatted at 2026-08-09 20:23:51 on dist-test-slave-rnrw"
server_key: "267b76bcff46e711dc374237b7c8af13"
server_key_iv: "9e07e50a6757ab4cac857c8907f0e2cb"
server_key_version: "encryptionkey@0"
I20260809 20:23:51.984459 15696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 20:23:51.988178 15712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:51.989765 15696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 20:23:51.989916 15696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "cf6d4316de894d77b0d167c9812a03eb"
format_stamp: "Formatted at 2026-08-09 20:23:51 on dist-test-slave-rnrw"
server_key: "267b76bcff46e711dc374237b7c8af13"
server_key_iv: "9e07e50a6757ab4cac857c8907f0e2cb"
server_key_version: "encryptionkey@0"
I20260809 20:23:51.990090 15696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:52.009869 15696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:52.014011 15696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:52.014263 15696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:52.033205 15696 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:45167
I20260809 20:23:52.033249 15766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:45167 every 8 connection(s)
I20260809 20:23:52.034673 15696 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:23:52.038571 15767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:52.043157 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 15696
I20260809 20:23:52.043428 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:23:52.044270 12600 external_mini_cluster.cc:1428] Setting key 0c515c96d56ccd3bf61d681d9de28539
I20260809 20:23:52.050314 15767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb: Bootstrap starting.
I20260809 20:23:52.053462 15767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:52.054469 15767 log.cc:824] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:52.057541 15767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb: No bootstrap required, opened a new log
Aug 09 20:23:52 dist-test-slave-rnrw krb5kdc[15683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:52.067286 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:52.046262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.12.78.62:45167)
0809 20:23:52.047095 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:52.047107 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:52.047146 (+ 39us) server_negotiation.cc:407] Connection header received
0809 20:23:52.049014 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:52.049037 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:52.049412 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:52.049783 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:52.051336 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:52.052760 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:52.054057 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:52.054778 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:52.054968 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:52.059694 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:52.059702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:52.059719 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:52.059764 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:52.062986 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:52.063881 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:52.063887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:52.063894 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:52.064067 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:52.064474 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:52.064477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:52.064479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:52.065134 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:52.065344 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:52.065845 (+ 501us) server_negotiation.cc:299] Negotiation successful
0809 20:23:52.066242 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":413,"threads_started":1}
I20260809 20:23:52.067219 15767 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6d4316de894d77b0d167c9812a03eb" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 45167 } }
I20260809 20:23:52.067672 15767 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:52.067751 15767 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6d4316de894d77b0d167c9812a03eb, State: Initialized, Role: FOLLOWER
I20260809 20:23:52.068253 15767 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6d4316de894d77b0d167c9812a03eb" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 45167 } }
I20260809 20:23:52.068439 15767 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:23:52.068532 15767 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:23:52.068801 15767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:52.071239 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:23:52.071429 15767 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6d4316de894d77b0d167c9812a03eb" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 45167 } }
I20260809 20:23:52.071971 15767 leader_election.cc:302] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf6d4316de894d77b0d167c9812a03eb; no voters:
I20260809 20:23:52.072662 15767 leader_election.cc:288] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:23:52.072840 15773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:52.073505 15773 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [term 1 LEADER]: Becoming Leader. State: Replica: cf6d4316de894d77b0d167c9812a03eb, State: Running, Role: LEADER
I20260809 20:23:52.074059 15773 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6d4316de894d77b0d167c9812a03eb" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 45167 } }
I20260809 20:23:52.075842 15773 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf6d4316de894d77b0d167c9812a03eb. Latest consensus state: current_term: 1 leader_uuid: "cf6d4316de894d77b0d167c9812a03eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6d4316de894d77b0d167c9812a03eb" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 45167 } } }
I20260809 20:23:52.076110 15773 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:52.076725 15775 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf6d4316de894d77b0d167c9812a03eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6d4316de894d77b0d167c9812a03eb" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 45167 } } }
I20260809 20:23:52.077121 15775 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:52.084733 15767 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:23:52.084988 15777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:23:52.090420 15777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:23:52.107471 15777 catalog_manager.cc:1347] Generated new cluster ID: 4eee3a7a97bb4a488cb0edc011bc254c
I20260809 20:23:52.107596 15777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:23:52.142076 15777 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:23:52.144668 15777 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:23:52.165287 15777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf6d4316de894d77b0d167c9812a03eb: Generated new TSK 0
I20260809 20:23:52.166232 15777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:23:52.172237 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:52.154695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.12.78.62:45167)
0809 20:23:52.154902 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:52.154913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:52.163295 (+ 8382us) server_negotiation.cc:407] Connection header received
0809 20:23:52.163520 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:52.163525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:52.163620 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:52.163769 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:52.165245 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:52.166547 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:52.168763 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:52.169233 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:52.169347 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:52.170691 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:52.170696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:52.170699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:23:52.171362 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:52.171441 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:23:52.171470 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:52.171575 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:52.171746 (+ 171us) server_negotiation.cc:299] Negotiation successful
0809 20:23:52.171839 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 20:23:52.189057 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 15696
I20260809 20:23:52.190934 15762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:52.284624 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 15696
2026-08-09T20:23:52Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2039 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:23:52 dist-test-slave-rnrw krb5kdc[15802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:23:52 dist-test-slave-rnrw krb5kdc[15802](info): set up 2 sockets
Aug 09 20:23:52 dist-test-slave-rnrw krb5kdc[15802](info): commencing operation
krb5kdc: starting...
W20260809 20:23:53.434111 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:23:53 dist-test-slave-rnrw krb5kdc[15802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:23:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:23:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:23:53.656941 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44695
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:44695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:53.987520 15815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:53.987841 15815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:53.988008 15815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:53.995450 15815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:23:53.995577 15815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:53.995664 15815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:53.995757 15815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:23:53.995841 15815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:23:54.006184 15815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:44695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:54.007721 15815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:54.009112 15815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:54.021515 15823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:54.021484 15826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:54.021541 15824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:54.022310 15815 server_base.cc:1034] running on GCE node
I20260809 20:23:54.023085 15815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:54.024457 15815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:54.025703 15815 hybrid_clock.cc:648] HybridClock initialized: now 1786307034025635 us; error 53 us; skew 500 ppm
I20260809 20:23:54.026091 15815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:54 dist-test-slave-rnrw krb5kdc[15802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307034, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:54.029397 15815 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:23:54.030858 15815 webserver.cc:460] Webserver started at http://127.12.78.62:41741/ using document root <none> and password file <none>
I20260809 20:23:54.031522 15815 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:54.031626 15815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:54.031894 15815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:54.034938 15815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "efd0f3d50d3b4ab8bc8fbbbea80f57d4"
format_stamp: "Formatted at 2026-08-09 20:23:54 on dist-test-slave-rnrw"
server_key: "c8a3396a891069a4f2aa8174f92247a7"
server_key_iv: "f4cb0ce15241e8959a2f9857fe7dad5a"
server_key_version: "encryptionkey@0"
I20260809 20:23:54.035667 15815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "efd0f3d50d3b4ab8bc8fbbbea80f57d4"
format_stamp: "Formatted at 2026-08-09 20:23:54 on dist-test-slave-rnrw"
server_key: "c8a3396a891069a4f2aa8174f92247a7"
server_key_iv: "f4cb0ce15241e8959a2f9857fe7dad5a"
server_key_version: "encryptionkey@0"
I20260809 20:23:54.041066 15815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 20:23:54.044366 15834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:54.045797 15815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:23:54.045989 15815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "efd0f3d50d3b4ab8bc8fbbbea80f57d4"
format_stamp: "Formatted at 2026-08-09 20:23:54 on dist-test-slave-rnrw"
server_key: "c8a3396a891069a4f2aa8174f92247a7"
server_key_iv: "f4cb0ce15241e8959a2f9857fe7dad5a"
server_key_version: "encryptionkey@0"
I20260809 20:23:54.046192 15815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:54.058806 15815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:54.063011 15815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:54.063315 15815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:54.082075 15815 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:44695
I20260809 20:23:54.082114 15888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:44695 every 8 connection(s)
I20260809 20:23:54.083534 15815 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:23:54.087298 15889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:23:54.088349 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 15815
I20260809 20:23:54.088636 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:23:54.089304 12600 external_mini_cluster.cc:1428] Setting key e2891340a33a438ed880ab5ed3086d8d
I20260809 20:23:54.102385 15889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4: Bootstrap starting.
Aug 09 20:23:54 dist-test-slave-rnrw krb5kdc[15802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:54.105696 15889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:54.107098 15889 log.cc:824] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:54.110597 15889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4: No bootstrap required, opened a new log
I20260809 20:23:54.113714 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:54.091404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48398 (local address 127.12.78.62:44695)
0809 20:23:54.092293 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:54.092309 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:54.092362 (+ 53us) server_negotiation.cc:407] Connection header received
0809 20:23:54.094317 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:54.094347 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:54.094888 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:54.095368 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:54.096849 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:54.098831 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:54.100108 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:54.101208 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:54.101398 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:54.105360 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:54.105372 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:54.105391 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:54.105439 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:54.109240 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:54.109913 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:54.109921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:54.109929 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:54.110140 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:54.110487 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:54.110492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:54.110496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:54.111228 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:54.111573 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:54.111823 (+ 250us) server_negotiation.cc:299] Negotiation successful
0809 20:23:54.112248 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":330,"threads_started":1}
I20260809 20:23:54.119648 15889 raft_consensus.cc:348] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd0f3d50d3b4ab8bc8fbbbea80f57d4" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44695 } }
I20260809 20:23:54.119966 15889 raft_consensus.cc:374] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:54.120046 15889 raft_consensus.cc:729] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd0f3d50d3b4ab8bc8fbbbea80f57d4, State: Initialized, Role: FOLLOWER
I20260809 20:23:54.120643 15889 consensus_queue.cc:260] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd0f3d50d3b4ab8bc8fbbbea80f57d4" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44695 } }
I20260809 20:23:54.120847 15889 raft_consensus.cc:388] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:23:54.120942 15889 raft_consensus.cc:482] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:23:54.121079 15889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:54.123224 15889 raft_consensus.cc:504] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd0f3d50d3b4ab8bc8fbbbea80f57d4" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44695 } }
I20260809 20:23:54.123653 15889 leader_election.cc:302] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efd0f3d50d3b4ab8bc8fbbbea80f57d4; no voters:
I20260809 20:23:54.124267 15889 leader_election.cc:288] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:23:54.124368 15894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:54.125597 15894 raft_consensus.cc:686] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [term 1 LEADER]: Becoming Leader. State: Replica: efd0f3d50d3b4ab8bc8fbbbea80f57d4, State: Running, Role: LEADER
I20260809 20:23:54.126168 15894 consensus_queue.cc:237] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd0f3d50d3b4ab8bc8fbbbea80f57d4" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44695 } }
I20260809 20:23:54.126497 15889 sys_catalog.cc:564] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:23:54.129189 15896 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efd0f3d50d3b4ab8bc8fbbbea80f57d4. Latest consensus state: current_term: 1 leader_uuid: "efd0f3d50d3b4ab8bc8fbbbea80f57d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd0f3d50d3b4ab8bc8fbbbea80f57d4" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44695 } } }
I20260809 20:23:54.129215 15895 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efd0f3d50d3b4ab8bc8fbbbea80f57d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd0f3d50d3b4ab8bc8fbbbea80f57d4" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44695 } } }
I20260809 20:23:54.129601 15896 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:54.129606 15895 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4 [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:54.131785 15901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:23:54.137372 15901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:23:54.145318 15901 catalog_manager.cc:1347] Generated new cluster ID: 31225da5f72f4b9b81ee2bc7e90b2494
I20260809 20:23:54.145440 15901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:23:54.174824 15901 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:23:54.176208 15901 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:23:54.195411 15901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efd0f3d50d3b4ab8bc8fbbbea80f57d4: Generated new TSK 0
I20260809 20:23:54.196389 15901 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:23:54.250736 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44695
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:54.593564 15915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:54.593926 15915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:54.594113 15915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:54.601609 15915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:54.601787 15915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:54.602006 15915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:23:54.612111 15915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:54.613626 15915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:54.615007 15915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:54.629889 15920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:54.629917 15923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:54.629837 15921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:54.631551 15915 server_base.cc:1034] running on GCE node
I20260809 20:23:54.632381 15915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:54.633529 15915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:54.634785 15915 hybrid_clock.cc:648] HybridClock initialized: now 1786307034634703 us; error 65 us; skew 500 ppm
I20260809 20:23:54.635150 15915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:54 dist-test-slave-rnrw krb5kdc[15802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307034, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:54.638676 15915 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:23:54.640254 15915 webserver.cc:460] Webserver started at http://127.12.78.1:40367/ using document root <none> and password file <none>
I20260809 20:23:54.641010 15915 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:54.641134 15915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:54.641400 15915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:54.644665 15915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "2b54d661197b4b48bd5105586f016515"
format_stamp: "Formatted at 2026-08-09 20:23:54 on dist-test-slave-rnrw"
server_key: "9b92b4a0f9b075185fca8698f5ca2416"
server_key_iv: "a5a829863518ee9f02631b136881f04c"
server_key_version: "encryptionkey@0"
I20260809 20:23:54.645455 15915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "2b54d661197b4b48bd5105586f016515"
format_stamp: "Formatted at 2026-08-09 20:23:54 on dist-test-slave-rnrw"
server_key: "9b92b4a0f9b075185fca8698f5ca2416"
server_key_iv: "a5a829863518ee9f02631b136881f04c"
server_key_version: "encryptionkey@0"
I20260809 20:23:54.651531 15915 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 20:23:54.655897 15930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:54.658142 15915 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 20:23:54.658353 15915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "2b54d661197b4b48bd5105586f016515"
format_stamp: "Formatted at 2026-08-09 20:23:54 on dist-test-slave-rnrw"
server_key: "9b92b4a0f9b075185fca8698f5ca2416"
server_key_iv: "a5a829863518ee9f02631b136881f04c"
server_key_version: "encryptionkey@0"
I20260809 20:23:54.658561 15915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:54.672472 15915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:54.676664 15915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:54.676926 15915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:54.677870 15915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:54.679230 15915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:54.679304 15915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:54.679388 15915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:54.679436 15915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:54.713914 15915 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:45447
I20260809 20:23:54.713999 16050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:45447 every 8 connection(s)
I20260809 20:23:54.715240 15915 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:23:54.724939 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 15915
I20260809 20:23:54.725188 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:23:54.725832 12600 external_mini_cluster.cc:1428] Setting key b1b89e8ad39a5f3275e0acb2dfe00e3c
Aug 09 20:23:54 dist-test-slave-rnrw krb5kdc[15802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307034, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:54.735508 16053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:54.717942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:55149 (local address 127.12.78.62:44695)
0809 20:23:54.718415 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:54.718420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:54.719253 (+ 833us) server_negotiation.cc:407] Connection header received
0809 20:23:54.721145 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:54.721150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:54.721243 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:54.721362 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:54.722891 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:54.723632 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:54.725293 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:54.725721 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:54.725872 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:54.729768 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:54.729775 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:54.729778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:54.729809 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:54.732388 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:54.733161 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:54.733171 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:54.733173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:54.733288 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:54.733731 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:54.733736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:54.733741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:54.734020 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:54.734181 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:54.734896 (+ 715us) server_negotiation.cc:299] Negotiation successful
0809 20:23:54.735134 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":260,"threads_started":1}
I20260809 20:23:54.736470 16054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:54.718350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44695 (local address 127.12.78.1:55149)
0809 20:23:54.719087 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:54.719125 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:54.720864 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:54.721615 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:54.721623 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:54.722139 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:54.722713 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:54.722730 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:54.723805 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:54.723809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:54.725147 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:54.725161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:54.725966 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:54.725976 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:54.726221 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:54.727002 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:54.727024 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:54.729533 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:54.732562 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:54.732574 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:54.732591 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:54.733027 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:54.733401 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:54.733406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:54.733410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:54.733609 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:54.734161 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:54.734169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:54.734522 (+ 353us) client_negotiation.cc:769] Sending connection context
0809 20:23:54.734782 (+ 260us) client_negotiation.cc:240] Negotiation successful
0809 20:23:54.735319 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":294,"threads_started":1}
I20260809 20:23:54.738270 16051 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44695
I20260809 20:23:54.738605 16051 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:54.739346 16051 heartbeater.cc:511] Master 127.12.78.62:44695 requested a full tablet report, sending...
I20260809 20:23:54.741730 15853 ts_manager.cc:194] Registered new tserver with Master: 2b54d661197b4b48bd5105586f016515 (127.12.78.1:45447)
I20260809 20:23:54.743986 15853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:55149
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:23:54.775821 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44695
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:55.115321 16057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:55.115655 16057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:55.115825 16057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:55.123185 16057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:55.123342 16057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:55.123529 16057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:23:55.133807 16057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:55.135133 16057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:55.136535 16057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:55.149507 16062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:55.149871 16063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:55.150034 16065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:55.151290 16057 server_base.cc:1034] running on GCE node
I20260809 20:23:55.151889 16057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:55.152946 16057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:55.154175 16057 hybrid_clock.cc:648] HybridClock initialized: now 1786307035154112 us; error 60 us; skew 500 ppm
I20260809 20:23:55.154495 16057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:55 dist-test-slave-rnrw krb5kdc[15802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307035, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:55.157718 16057 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:23:55.159317 16057 webserver.cc:460] Webserver started at http://127.12.78.2:46811/ using document root <none> and password file <none>
I20260809 20:23:55.160041 16057 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:55.160205 16057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:55.160586 16057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:55.163749 16057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "82ac089a8a4846a3a1c4661213da7316"
format_stamp: "Formatted at 2026-08-09 20:23:55 on dist-test-slave-rnrw"
server_key: "c8887ed5bb19607440d236047a65443f"
server_key_iv: "b44c677abf1c688659be0cd69f2cc1c1"
server_key_version: "encryptionkey@0"
I20260809 20:23:55.164601 16057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "82ac089a8a4846a3a1c4661213da7316"
format_stamp: "Formatted at 2026-08-09 20:23:55 on dist-test-slave-rnrw"
server_key: "c8887ed5bb19607440d236047a65443f"
server_key_iv: "b44c677abf1c688659be0cd69f2cc1c1"
server_key_version: "encryptionkey@0"
I20260809 20:23:55.170063 16057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 20:23:55.173657 16072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:55.175022 16057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 20:23:55.175211 16057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "82ac089a8a4846a3a1c4661213da7316"
format_stamp: "Formatted at 2026-08-09 20:23:55 on dist-test-slave-rnrw"
server_key: "c8887ed5bb19607440d236047a65443f"
server_key_iv: "b44c677abf1c688659be0cd69f2cc1c1"
server_key_version: "encryptionkey@0"
I20260809 20:23:55.175416 16057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:55.205838 16057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:55.210108 16057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:55.210433 16057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:55.211441 16057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:55.212886 16057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:55.212990 16057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:55.213177 16057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:55.213263 16057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:55.248327 16057 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:45605
I20260809 20:23:55.248364 16193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:45605 every 8 connection(s)
I20260809 20:23:55.249747 16057 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:23:55.258859 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 16057
I20260809 20:23:55.259071 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:23:55.259498 12600 external_mini_cluster.cc:1428] Setting key e2a254ff91334a5e6af81c2e504f6e15
Aug 09 20:23:55 dist-test-slave-rnrw krb5kdc[15802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307035, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:55.271311 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:55.252455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:60267 (local address 127.12.78.62:44695)
0809 20:23:55.252933 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:55.252937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:55.253893 (+ 956us) server_negotiation.cc:407] Connection header received
0809 20:23:55.255785 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:55.255790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:55.255881 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:55.256005 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:55.257560 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.258279 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.260072 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.260737 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.260877 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:55.265047 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:55.265053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:55.265056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:55.265094 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:55.268049 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:55.268886 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:55.268892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:55.268895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:55.269042 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:55.269511 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:55.269521 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:55.269523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:55.269791 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:55.269965 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:55.270672 (+ 707us) server_negotiation.cc:299] Negotiation successful
0809 20:23:55.270911 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":377,"thread_start_us":270,"threads_started":1}
I20260809 20:23:55.272356 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:55.252967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44695 (local address 127.12.78.2:60267)
0809 20:23:55.253725 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:55.253772 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:55.255512 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:55.256244 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:55.256254 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:55.256789 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:55.257365 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:55.257383 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.258446 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.258458 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:55.259907 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:55.259920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.260991 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.260999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:55.261304 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:55.262068 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:55.262092 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:55.264781 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:55.268211 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:55.268222 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:55.268240 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:55.268754 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:55.269166 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:55.269172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:55.269175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:55.269381 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:55.269922 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:55.269930 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:55.270297 (+ 367us) client_negotiation.cc:769] Sending connection context
0809 20:23:55.270563 (+ 266us) client_negotiation.cc:240] Negotiation successful
0809 20:23:55.271163 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":328,"threads_started":1}
I20260809 20:23:55.273978 16194 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44695
I20260809 20:23:55.274369 16194 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:55.275321 16194 heartbeater.cc:511] Master 127.12.78.62:44695 requested a full tablet report, sending...
I20260809 20:23:55.277235 15853 ts_manager.cc:194] Registered new tserver with Master: 82ac089a8a4846a3a1c4661213da7316 (127.12.78.2:45605)
I20260809 20:23:55.278596 15853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:60267
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:23:55.311520 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44695
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:55.659003 16200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:55.659302 16200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:55.659397 16200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:55.667065 16200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:55.667204 16200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:55.667321 16200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:23:55.677415 16200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:55.678754 16200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:55.680162 16200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:55.694475 16205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:55.694456 16206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:55.694619 16208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:55.695386 16200 server_base.cc:1034] running on GCE node
I20260809 20:23:55.696175 16200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:55.697345 16200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:55.698640 16200 hybrid_clock.cc:648] HybridClock initialized: now 1786307035698590 us; error 45 us; skew 500 ppm
I20260809 20:23:55.699023 16200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:55 dist-test-slave-rnrw krb5kdc[15802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307035, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:55.708796 16200 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:23:55.710574 16200 webserver.cc:460] Webserver started at http://127.12.78.3:37097/ using document root <none> and password file <none>
I20260809 20:23:55.711381 16200 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:55.711560 16200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:55.711901 16200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:55.715116 16200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "a3a6e3e161c149c98674be2dbdc5b0af"
format_stamp: "Formatted at 2026-08-09 20:23:55 on dist-test-slave-rnrw"
server_key: "53d33de01096209a462416f41f47c547"
server_key_iv: "b05ea72595861d04d7a49be947713ccd"
server_key_version: "encryptionkey@0"
I20260809 20:23:55.715943 16200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "a3a6e3e161c149c98674be2dbdc5b0af"
format_stamp: "Formatted at 2026-08-09 20:23:55 on dist-test-slave-rnrw"
server_key: "53d33de01096209a462416f41f47c547"
server_key_iv: "b05ea72595861d04d7a49be947713ccd"
server_key_version: "encryptionkey@0"
I20260809 20:23:55.721403 16200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 20:23:55.725248 16216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:55.726732 16200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:23:55.726946 16200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "a3a6e3e161c149c98674be2dbdc5b0af"
format_stamp: "Formatted at 2026-08-09 20:23:55 on dist-test-slave-rnrw"
server_key: "53d33de01096209a462416f41f47c547"
server_key_iv: "b05ea72595861d04d7a49be947713ccd"
server_key_version: "encryptionkey@0"
I20260809 20:23:55.727159 16200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:55.746137 16200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:55.747177 16051 heartbeater.cc:503] Master 127.12.78.62:44695 was elected leader, sending a full tablet report...
I20260809 20:23:55.751155 16200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:55.751430 16200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:55.752398 16200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:55.753852 16200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:55.753929 16200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:55.754017 16200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:55.754065 16200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:55.788978 16200 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:44053
I20260809 20:23:55.788997 16341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:44053 every 8 connection(s)
I20260809 20:23:55.790338 16200 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:23:55.794175 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 16200
I20260809 20:23:55.794394 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:23:55.794907 12600 external_mini_cluster.cc:1428] Setting key 79f917ca3abc0ab06c0e3cde356def6d
Aug 09 20:23:55 dist-test-slave-rnrw krb5kdc[15802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307035, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:55.811583 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:55.793104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:50583 (local address 127.12.78.62:44695)
0809 20:23:55.793561 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:55.793567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:55.794744 (+ 1177us) server_negotiation.cc:407] Connection header received
0809 20:23:55.797059 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:55.797064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:55.797140 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:55.797230 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:55.798948 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.799738 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.801336 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.801762 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.801860 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:55.805255 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:55.805260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:55.805263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:55.805296 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:55.807992 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:55.808823 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:55.808848 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:55.808853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:55.809011 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:55.809616 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:55.809621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:55.809624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:55.810007 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:55.810158 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:55.811135 (+ 977us) server_negotiation.cc:299] Negotiation successful
0809 20:23:55.811311 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":199,"threads_started":1}
I20260809 20:23:55.812640 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:55.793713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44695 (local address 127.12.78.3:50583)
0809 20:23:55.794559 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:55.794606 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:55.796780 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:55.797507 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:55.797520 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:55.798242 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:55.798799 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:55.798815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.799873 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.799877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:55.801207 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:55.801219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.801951 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.801955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:55.802156 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:55.802771 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:55.802798 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:55.805051 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:55.808133 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:55.808141 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:55.808153 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:55.808688 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:55.809163 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:55.809168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:55.809171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:55.809453 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:55.810197 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:55.810205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:55.810733 (+ 528us) client_negotiation.cc:769] Sending connection context
0809 20:23:55.811021 (+ 288us) client_negotiation.cc:240] Negotiation successful
0809 20:23:55.811439 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":374,"threads_started":1}
I20260809 20:23:55.814075 16342 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44695
I20260809 20:23:55.814404 16342 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:55.815214 16342 heartbeater.cc:511] Master 127.12.78.62:44695 requested a full tablet report, sending...
I20260809 20:23:55.816746 15853 ts_manager.cc:194] Registered new tserver with Master: a3a6e3e161c149c98674be2dbdc5b0af (127.12.78.3:44053)
I20260809 20:23:55.817523 15853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:50583
I20260809 20:23:55.823329 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 20:23:55 dist-test-slave-rnrw krb5kdc[15802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307035, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 20:23:55 dist-test-slave-rnrw krb5kdc[15802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307035, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:55.877008 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:55.862647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48402 (local address 127.12.78.62:44695)
0809 20:23:55.862910 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:55.862919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:55.863273 (+ 354us) server_negotiation.cc:407] Connection header received
0809 20:23:55.863514 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:55.863526 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:55.863598 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:55.863743 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:55.865325 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.866080 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.867348 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.867814 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.867925 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:55.871946 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:55.871952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:55.871955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:55.871992 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:55.874483 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:55.875121 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:55.875124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:55.875127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:55.875242 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:55.875598 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:55.875601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:55.875603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:55.875940 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:55.876128 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:55.876386 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 20:23:55.876678 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260809 20:23:55.877616 15853 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48402
I20260809 20:23:55.879292 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 15915
Aug 09 20:23:55 dist-test-slave-rnrw krb5kdc[15802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307035, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:23:55.896104 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:55.881215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.12.78.1:45447)
0809 20:23:55.881909 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:55.881916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:55.881943 (+ 27us) server_negotiation.cc:407] Connection header received
0809 20:23:55.882009 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:55.882016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:55.882225 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:55.882350 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:55.883525 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.884592 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.885765 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.886389 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.886493 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:55.890513 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:55.890525 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:55.890542 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:55.890588 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:55.893408 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:55.894052 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:55.894059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:55.894061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:55.894179 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:55.894483 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:55.894486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:55.894488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:55.894987 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:55.895237 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:55.895480 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 20:23:55.895677 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":380,"threads_started":1}
W20260809 20:23:55.897492 16046 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45618
W20260809 20:23:55.898591 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 15915: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:23:55.898808 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 15915
I20260809 20:23:55.916894 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 16057
Aug 09 20:23:55 dist-test-slave-rnrw krb5kdc[15802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307035, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:23:55.933416 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:55.917923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.12.78.2:45605)
0809 20:23:55.918516 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:55.918523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:55.918553 (+ 30us) server_negotiation.cc:407] Connection header received
0809 20:23:55.918615 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:55.918621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:55.918841 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:55.918981 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:55.920294 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.921428 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.922734 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.923369 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.923486 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:55.927822 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:55.927836 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:55.927852 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:55.927889 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:55.930732 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:55.931361 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:55.931368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:55.931370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:55.931487 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:55.931854 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:55.931857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:55.931859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:55.932253 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:55.932485 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:55.932841 (+ 356us) server_negotiation.cc:299] Negotiation successful
0809 20:23:55.933062 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":305,"threads_started":1}
W20260809 20:23:55.934703 16189 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56854
W20260809 20:23:55.935771 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 16057: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:23:55.935951 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 16057
I20260809 20:23:55.954346 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 16200
Aug 09 20:23:55 dist-test-slave-rnrw krb5kdc[15802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307035, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:23:55.971848 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:55.955337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.12.78.3:44053)
0809 20:23:55.955998 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:55.956005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:55.956035 (+ 30us) server_negotiation.cc:407] Connection header received
0809 20:23:55.956097 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:55.956104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:55.956314 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:55.956449 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:55.957895 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.958967 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.960266 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:55.960946 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:55.961050 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:55.965473 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:55.965487 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:55.965502 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:55.965537 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:55.969098 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:55.969756 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:55.969763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:55.969766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:55.969893 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:55.970232 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:55.970235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:55.970237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:55.970592 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:55.970829 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:55.971213 (+ 384us) server_negotiation.cc:299] Negotiation successful
0809 20:23:55.971469 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":325,"threads_started":1}
W20260809 20:23:55.973238 16337 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35102
W20260809 20:23:55.974370 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 16200: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:23:55.974502 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 16200
I20260809 20:23:55.992492 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 15815
I20260809 20:23:55.993826 15884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:56.098721 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 15815
2026-08-09T20:23:56Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3820 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:23:56 dist-test-slave-rnrw krb5kdc[16361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:23:56 dist-test-slave-rnrw krb5kdc[16361](info): set up 2 sockets
Aug 09 20:23:56 dist-test-slave-rnrw krb5kdc[16361](info): commencing operation
krb5kdc: starting...
W20260809 20:23:57.230012 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:23:57 dist-test-slave-rnrw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:23:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:23:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:23:57.448282 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:33583
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:33583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:57.772848 16375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:57.773198 16375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:57.773401 16375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:57.780553 16375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:23:57.780711 16375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:57.780819 16375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:57.780921 16375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:23:57.780998 16375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:23:57.790700 16375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:33583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:33583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:57.792032 16375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:57.793361 16375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:57.805127 16380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:57.805055 16384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:57.805104 16381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:57.806735 16375 server_base.cc:1034] running on GCE node
I20260809 20:23:57.807511 16375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:57.808948 16375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:57.810221 16375 hybrid_clock.cc:648] HybridClock initialized: now 1786307037810174 us; error 50 us; skew 500 ppm
I20260809 20:23:57.810621 16375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:57 dist-test-slave-rnrw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307037, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:57.813834 16375 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:23:57.815378 16375 webserver.cc:460] Webserver started at http://127.12.78.62:35007/ using document root <none> and password file <none>
I20260809 20:23:57.816092 16375 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:57.816242 16375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:57.816638 16375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:57.819618 16375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "44ce372274e14c01b711953e40a2ebc9"
format_stamp: "Formatted at 2026-08-09 20:23:57 on dist-test-slave-rnrw"
server_key: "3ca2344706db0430a530b7e91687b116"
server_key_iv: "50a9dfa23c28cabb7b3796a1ebb3a7bf"
server_key_version: "encryptionkey@0"
I20260809 20:23:57.820447 16375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "44ce372274e14c01b711953e40a2ebc9"
format_stamp: "Formatted at 2026-08-09 20:23:57 on dist-test-slave-rnrw"
server_key: "3ca2344706db0430a530b7e91687b116"
server_key_iv: "50a9dfa23c28cabb7b3796a1ebb3a7bf"
server_key_version: "encryptionkey@0"
I20260809 20:23:57.825583 16375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 20:23:57.829048 16391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:57.830397 16375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 20:23:57.830618 16375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "44ce372274e14c01b711953e40a2ebc9"
format_stamp: "Formatted at 2026-08-09 20:23:57 on dist-test-slave-rnrw"
server_key: "3ca2344706db0430a530b7e91687b116"
server_key_iv: "50a9dfa23c28cabb7b3796a1ebb3a7bf"
server_key_version: "encryptionkey@0"
I20260809 20:23:57.830827 16375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:57.870581 16375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:57.875099 16375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:57.875416 16375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:57.895094 16375 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:33583
I20260809 20:23:57.895138 16445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:33583 every 8 connection(s)
I20260809 20:23:57.896596 16375 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:23:57.899210 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 16375
I20260809 20:23:57.899437 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:23:57.900267 12600 external_mini_cluster.cc:1428] Setting key 16881e6d2cf12e1a8f1a9dc33cad9b3c
I20260809 20:23:57.901242 16446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 20:23:57 dist-test-slave-rnrw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:57.916265 16446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9: Bootstrap starting.
I20260809 20:23:57.919559 16446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9: Neither blocks nor log segments found. Creating new log.
I20260809 20:23:57.920640 16446 log.cc:824] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9: Log is configured to *not* fsync() on all Append() calls
I20260809 20:23:57.923497 16449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:57.902486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.12.78.62:33583)
0809 20:23:57.903296 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:57.903307 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:57.903347 (+ 40us) server_negotiation.cc:407] Connection header received
0809 20:23:57.904957 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:57.904980 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:57.905368 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:57.905714 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:57.907628 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:57.909053 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:57.910786 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:57.911437 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:57.911571 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:57.916197 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:57.916205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:57.916221 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:57.916260 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:57.919439 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:57.920213 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:57.920223 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:57.920235 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:57.920397 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:57.920809 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:57.920816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:57.920819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:57.921336 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:57.921687 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:57.922097 (+ 410us) server_negotiation.cc:299] Negotiation successful
0809 20:23:57.922513 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":362,"threads_started":1}
I20260809 20:23:57.923660 16446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9: No bootstrap required, opened a new log
I20260809 20:23:57.931900 16446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ce372274e14c01b711953e40a2ebc9" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33583 } }
I20260809 20:23:57.932422 16446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:23:57.932582 16446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ce372274e14c01b711953e40a2ebc9, State: Initialized, Role: FOLLOWER
I20260809 20:23:57.933238 16446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ce372274e14c01b711953e40a2ebc9" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33583 } }
I20260809 20:23:57.933496 16446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:23:57.933656 16446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:23:57.933835 16446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:23:57.936335 16446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ce372274e14c01b711953e40a2ebc9" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33583 } }
I20260809 20:23:57.936964 16446 leader_election.cc:302] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44ce372274e14c01b711953e40a2ebc9; no voters:
I20260809 20:23:57.937601 16446 leader_election.cc:288] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:23:57.937784 16451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:23:57.938450 16451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [term 1 LEADER]: Becoming Leader. State: Replica: 44ce372274e14c01b711953e40a2ebc9, State: Running, Role: LEADER
I20260809 20:23:57.939016 16451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ce372274e14c01b711953e40a2ebc9" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33583 } }
I20260809 20:23:57.939788 16446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:23:57.942003 16454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44ce372274e14c01b711953e40a2ebc9. Latest consensus state: current_term: 1 leader_uuid: "44ce372274e14c01b711953e40a2ebc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ce372274e14c01b711953e40a2ebc9" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33583 } } }
I20260809 20:23:57.942073 16452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44ce372274e14c01b711953e40a2ebc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ce372274e14c01b711953e40a2ebc9" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33583 } } }
I20260809 20:23:57.942344 16454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:57.942399 16452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9 [sys.catalog]: This master's current role is: LEADER
I20260809 20:23:57.945360 16460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:23:57.950827 16460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:23:57.959226 16460 catalog_manager.cc:1347] Generated new cluster ID: e04850d98f964baf996d102b13c3fcce
I20260809 20:23:57.959347 16460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:23:57.974471 16460 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:23:57.976037 16460 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:23:57.988059 16460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44ce372274e14c01b711953e40a2ebc9: Generated new TSK 0
I20260809 20:23:57.989061 16460 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:23:58.048950 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33583
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:58.391556 16474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:58.391899 16474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:58.392059 16474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:58.399297 16474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:58.399457 16474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:58.399645 16474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:23:58.409854 16474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:58.411177 16474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:58.412482 16474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:58.426376 16480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:58.426430 16483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:58.426451 16481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:58.427417 16474 server_base.cc:1034] running on GCE node
I20260809 20:23:58.428202 16474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:58.429493 16474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:58.430732 16474 hybrid_clock.cc:648] HybridClock initialized: now 1786307038430674 us; error 51 us; skew 500 ppm
I20260809 20:23:58.431046 16474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:58 dist-test-slave-rnrw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307038, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:58.434218 16474 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:23:58.435776 16474 webserver.cc:460] Webserver started at http://127.12.78.1:38115/ using document root <none> and password file <none>
I20260809 20:23:58.436483 16474 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:58.436662 16474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:58.436982 16474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:58.440047 16474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "66d92651edb6457da7f8248f9e1f6b3c"
format_stamp: "Formatted at 2026-08-09 20:23:58 on dist-test-slave-rnrw"
server_key: "0af0f0243bb7b583577da167d2e295c6"
server_key_iv: "f477aa3b22c33761124578c4bd496729"
server_key_version: "encryptionkey@0"
I20260809 20:23:58.440996 16474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "66d92651edb6457da7f8248f9e1f6b3c"
format_stamp: "Formatted at 2026-08-09 20:23:58 on dist-test-slave-rnrw"
server_key: "0af0f0243bb7b583577da167d2e295c6"
server_key_iv: "f477aa3b22c33761124578c4bd496729"
server_key_version: "encryptionkey@0"
I20260809 20:23:58.446539 16474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:23:58.450172 16490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:58.451574 16474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 20:23:58.451761 16474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "66d92651edb6457da7f8248f9e1f6b3c"
format_stamp: "Formatted at 2026-08-09 20:23:58 on dist-test-slave-rnrw"
server_key: "0af0f0243bb7b583577da167d2e295c6"
server_key_iv: "f477aa3b22c33761124578c4bd496729"
server_key_version: "encryptionkey@0"
I20260809 20:23:58.451977 16474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:58.462527 16474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:58.466919 16474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:58.467330 16474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:58.468318 16474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:58.469738 16474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:58.469843 16474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:58.469991 16474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:58.470067 16474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:58.504854 16474 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:44549
I20260809 20:23:58.504922 16609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:44549 every 8 connection(s)
I20260809 20:23:58.506551 16474 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:23:58.512249 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 16474
I20260809 20:23:58.512456 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:23:58.513139 12600 external_mini_cluster.cc:1428] Setting key 20dada0e119d9fa97d578b4df8c8bfec
Aug 09 20:23:58 dist-test-slave-rnrw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307038, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:58.527887 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:58.509802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:49305 (local address 127.12.78.62:33583)
0809 20:23:58.510288 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:58.510295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:58.511213 (+ 918us) server_negotiation.cc:407] Connection header received
0809 20:23:58.513230 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:58.513236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:58.513350 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:58.513471 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:58.515185 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:58.515915 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:58.517524 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:58.517933 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:58.518069 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:58.521568 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:58.521573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:58.521576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:58.521608 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:58.524192 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:58.525189 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:58.525193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:58.525195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:58.525319 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:58.525829 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:58.525833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:58.525835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:58.526098 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:58.526209 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:58.527442 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0809 20:23:58.527613 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":250,"threads_started":1}
I20260809 20:23:58.529513 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:58.510242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:33583 (local address 127.12.78.1:49305)
0809 20:23:58.511043 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:58.511084 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:58.512958 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:58.513744 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:58.513756 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:58.514343 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:58.515006 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:58.515033 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:58.516075 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:58.516080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:58.517364 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:58.517376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:58.518168 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:58.518177 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:58.518407 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:58.519103 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:58.519127 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:58.521340 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:58.524339 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:58.524351 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:58.524370 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:58.525067 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:58.525427 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:58.525434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:58.525440 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:58.525701 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:58.526241 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:58.526252 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:58.527005 (+ 753us) client_negotiation.cc:769] Sending connection context
0809 20:23:58.527339 (+ 334us) client_negotiation.cc:240] Negotiation successful
0809 20:23:58.527934 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":337,"threads_started":1}
I20260809 20:23:58.531140 16610 heartbeater.cc:347] Connected to a master server at 127.12.78.62:33583
I20260809 20:23:58.531458 16610 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:58.532160 16610 heartbeater.cc:511] Master 127.12.78.62:33583 requested a full tablet report, sending...
I20260809 20:23:58.534294 16409 ts_manager.cc:194] Registered new tserver with Master: 66d92651edb6457da7f8248f9e1f6b3c (127.12.78.1:44549)
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
I20260809 20:23:58.535934 16409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:49305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:23:58.562160 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33583
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:58.894750 16616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:58.895138 16616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:58.895308 16616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:58.906783 16616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:58.906929 16616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:58.907104 16616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:23:58.920332 16616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:58.921626 16616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:58.922901 16616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:58.936430 16622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:58.936461 16623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:58.936712 16625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:58.937150 16616 server_base.cc:1034] running on GCE node
I20260809 20:23:58.937752 16616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:58.939064 16616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:58.940287 16616 hybrid_clock.cc:648] HybridClock initialized: now 1786307038940239 us; error 40 us; skew 500 ppm
I20260809 20:23:58.940677 16616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:58 dist-test-slave-rnrw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307038, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:58.943848 16616 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:23:58.945473 16616 webserver.cc:460] Webserver started at http://127.12.78.2:36469/ using document root <none> and password file <none>
I20260809 20:23:58.946161 16616 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:58.946306 16616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:58.946621 16616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:58.949681 16616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "dd4b4867bb01492eafc09574034b2ea0"
format_stamp: "Formatted at 2026-08-09 20:23:58 on dist-test-slave-rnrw"
server_key: "e1428a7bab2b38d8571ee1cac5f53c1f"
server_key_iv: "2c5d7eac0eb25ed86c1712fa4c8a9d47"
server_key_version: "encryptionkey@0"
I20260809 20:23:58.950480 16616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "dd4b4867bb01492eafc09574034b2ea0"
format_stamp: "Formatted at 2026-08-09 20:23:58 on dist-test-slave-rnrw"
server_key: "e1428a7bab2b38d8571ee1cac5f53c1f"
server_key_iv: "2c5d7eac0eb25ed86c1712fa4c8a9d47"
server_key_version: "encryptionkey@0"
I20260809 20:23:58.955729 16616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:23:58.959285 16632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:58.960675 16616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:23:58.960829 16616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "dd4b4867bb01492eafc09574034b2ea0"
format_stamp: "Formatted at 2026-08-09 20:23:58 on dist-test-slave-rnrw"
server_key: "e1428a7bab2b38d8571ee1cac5f53c1f"
server_key_iv: "2c5d7eac0eb25ed86c1712fa4c8a9d47"
server_key_version: "encryptionkey@0"
I20260809 20:23:58.961028 16616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:58.975734 16616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:58.979964 16616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:58.980264 16616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:58.981307 16616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:58.982636 16616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:58.982717 16616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:58.982889 16616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:58.982944 16616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:59.017489 16616 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:34681
I20260809 20:23:59.017583 16749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:34681 every 8 connection(s)
I20260809 20:23:59.018872 16616 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:23:59.023144 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 16616
I20260809 20:23:59.023348 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:23:59.023844 12600 external_mini_cluster.cc:1428] Setting key cb68a051810112f27d34cbe0efdf1635
Aug 09 20:23:59 dist-test-slave-rnrw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307038, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:59.039554 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:59.021760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:60639 (local address 127.12.78.62:33583)
0809 20:23:59.021954 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:59.021959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:59.023318 (+ 1359us) server_negotiation.cc:407] Connection header received
0809 20:23:59.025352 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:59.025358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:59.025451 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:59.025565 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:59.027057 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.027786 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.029406 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.029844 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.029987 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:59.033638 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:59.033643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:59.033646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:59.033678 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:59.036481 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.037257 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.037262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.037267 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.037401 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.037849 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.037853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.037855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.038128 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:59.038285 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:59.038976 (+ 691us) server_negotiation.cc:299] Negotiation successful
0809 20:23:59.039211 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 20:23:59.040742 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:59.022274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:33583 (local address 127.12.78.2:60639)
0809 20:23:59.023110 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:59.023191 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:59.025076 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:59.025795 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:59.025803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:59.026318 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:59.026891 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:59.026907 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.027953 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.027963 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:59.029252 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:59.029263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.030090 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.030097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:59.030361 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:59.031142 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:59.031166 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:59.033405 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:59.036680 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:59.036689 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:59.036706 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:59.037128 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:59.037517 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:59.037523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:59.037526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:59.037718 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:59.038258 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:59.038266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:59.038621 (+ 355us) client_negotiation.cc:769] Sending connection context
0809 20:23:59.038871 (+ 250us) client_negotiation.cc:240] Negotiation successful
0809 20:23:59.039384 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":301,"threads_started":1}
I20260809 20:23:59.042387 16750 heartbeater.cc:347] Connected to a master server at 127.12.78.62:33583
I20260809 20:23:59.042704 16750 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:59.043412 16750 heartbeater.cc:511] Master 127.12.78.62:33583 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
I20260809 20:23:59.045094 16409 ts_manager.cc:194] Registered new tserver with Master: dd4b4867bb01492eafc09574034b2ea0 (127.12.78.2:34681)
I20260809 20:23:59.045972 16409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:60639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:23:59.072418 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33583
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:23:59.411150 16755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:23:59.411429 16755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:23:59.411530 16755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:23:59.418789 16755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:23:59.418910 16755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:23:59.419028 16755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:23:59.429678 16755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:23:59.431094 16755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:23:59.432468 16755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:23:59.446290 16764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:59.446244 16762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:23:59.446338 16761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:23:59.447198 16755 server_base.cc:1034] running on GCE node
I20260809 20:23:59.447836 16755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:23:59.448860 16755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:23:59.450098 16755 hybrid_clock.cc:648] HybridClock initialized: now 1786307039450044 us; error 48 us; skew 500 ppm
I20260809 20:23:59.450430 16755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:23:59 dist-test-slave-rnrw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307039, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:23:59.453799 16755 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:23:59.455305 16755 webserver.cc:460] Webserver started at http://127.12.78.3:44453/ using document root <none> and password file <none>
I20260809 20:23:59.456064 16755 fs_manager.cc:359] Metadata directory not provided
I20260809 20:23:59.456333 16755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:23:59.456780 16755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:23:59.459785 16755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "453c92da4a6d4765b4b3b57d9bba8658"
format_stamp: "Formatted at 2026-08-09 20:23:59 on dist-test-slave-rnrw"
server_key: "1668ff0859b654b37ef09453b434cb17"
server_key_iv: "7f3dd9ec4d107e22f3190f1038d6ec3d"
server_key_version: "encryptionkey@0"
I20260809 20:23:59.460667 16755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "453c92da4a6d4765b4b3b57d9bba8658"
format_stamp: "Formatted at 2026-08-09 20:23:59 on dist-test-slave-rnrw"
server_key: "1668ff0859b654b37ef09453b434cb17"
server_key_iv: "7f3dd9ec4d107e22f3190f1038d6ec3d"
server_key_version: "encryptionkey@0"
I20260809 20:23:59.466202 16755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 20:23:59.469859 16772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:59.471249 16755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:23:59.471436 16755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "453c92da4a6d4765b4b3b57d9bba8658"
format_stamp: "Formatted at 2026-08-09 20:23:59 on dist-test-slave-rnrw"
server_key: "1668ff0859b654b37ef09453b434cb17"
server_key_iv: "7f3dd9ec4d107e22f3190f1038d6ec3d"
server_key_version: "encryptionkey@0"
I20260809 20:23:59.471653 16755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:23:59.497954 16755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:23:59.502607 16755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:23:59.502933 16755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:23:59.504048 16755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:23:59.505578 16755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:23:59.505663 16755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:59.505795 16755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:23:59.505847 16755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:23:59.539155 16610 heartbeater.cc:503] Master 127.12.78.62:33583 was elected leader, sending a full tablet report...
I20260809 20:23:59.539546 16755 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:45193
I20260809 20:23:59.539587 16891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:45193 every 8 connection(s)
I20260809 20:23:59.541304 16755 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:23:59.544898 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 16755
I20260809 20:23:59.545091 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:23:59.545620 12600 external_mini_cluster.cc:1428] Setting key 3c42d522739c7e9954dabe799e1ee13d
Aug 09 20:23:59 dist-test-slave-rnrw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307039, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:59.562017 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:59.543918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:50855 (local address 127.12.78.62:33583)
0809 20:23:59.544367 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:59.544374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:59.545782 (+ 1408us) server_negotiation.cc:407] Connection header received
0809 20:23:59.547889 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:59.547893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:59.547964 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:59.548052 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:59.549683 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.550446 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.551922 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.552314 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.552405 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:59.555774 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:59.555779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:59.555782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:59.555813 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:59.558517 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.559410 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.559417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.559420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.559572 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.560148 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.560152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.560154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.560476 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:59.560695 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:59.561540 (+ 845us) server_negotiation.cc:299] Negotiation successful
0809 20:23:59.561730 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":216,"threads_started":1}
I20260809 20:23:59.562983 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:59.544701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:33583 (local address 127.12.78.3:50855)
0809 20:23:59.545612 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0809 20:23:59.545666 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 20:23:59.547575 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:23:59.548331 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:23:59.548344 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:23:59.548974 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:23:59.549531 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:59.549548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.550579 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.550583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:59.551799 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:23:59.551810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.552493 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.552498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:23:59.552761 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:59.553375 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:23:59.553396 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:23:59.555555 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:23:59.558671 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:59.558681 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:59.558696 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:59.559253 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:59.559705 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:23:59.559710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:23:59.559713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:23:59.560001 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:23:59.560713 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:23:59.560721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:23:59.561169 (+ 448us) client_negotiation.cc:769] Sending connection context
0809 20:23:59.561434 (+ 265us) client_negotiation.cc:240] Negotiation successful
0809 20:23:59.561829 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":420,"threads_started":1}
I20260809 20:23:59.564335 16892 heartbeater.cc:347] Connected to a master server at 127.12.78.62:33583
I20260809 20:23:59.564749 16892 heartbeater.cc:464] Registering TS with master...
I20260809 20:23:59.565456 16892 heartbeater.cc:511] Master 127.12.78.62:33583 requested a full tablet report, sending...
I20260809 20:23:59.566834 16409 ts_manager.cc:194] Registered new tserver with Master: 453c92da4a6d4765b4b3b57d9bba8658 (127.12.78.3:45193)
I20260809 20:23:59.567649 16409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:50855
I20260809 20:23:59.573351 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 20:23:59 dist-test-slave-rnrw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 20:23:59 dist-test-slave-rnrw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:23:59.627142 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:59.611106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.12.78.1:44549)
0809 20:23:59.612054 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:59.612061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:59.612086 (+ 25us) server_negotiation.cc:407] Connection header received
0809 20:23:59.612148 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:59.612154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:59.612371 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:59.612593 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:59.614085 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.615117 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.616371 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.617040 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.617188 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:59.621249 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:59.621260 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:59.621275 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:59.621317 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:59.624161 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.624893 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.624901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.624903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.625049 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.625451 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.625455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.625457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.625818 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:59.626147 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:59.626474 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 20:23:59.626776 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":544,"threads_started":1}
Aug 09 20:23:59 dist-test-slave-rnrw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:23:59.646345 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:59.631598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.12.78.62:33583)
0809 20:23:59.631867 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:59.631875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:59.632064 (+ 189us) server_negotiation.cc:407] Connection header received
0809 20:23:59.632266 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:59.632272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:59.632351 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:59.632457 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:59.633923 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.634764 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.636109 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.636590 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.636737 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:59.640937 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:59.640942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:59.640946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:59.640982 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:59.643740 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.644468 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.644472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.644474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.644680 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.645071 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.645074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.645076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.645407 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:59.645593 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:59.645903 (+ 310us) server_negotiation.cc:299] Negotiation successful
0809 20:23:59.646087 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260809 20:23:59.647007 16409 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35510
I20260809 20:23:59.648088 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 16474
I20260809 20:23:59.661171 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:59.649024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35478 (local address 127.12.78.1:44549)
0809 20:23:59.649253 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:59.649260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:59.649532 (+ 272us) server_negotiation.cc:407] Connection header received
0809 20:23:59.649646 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:59.649649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:59.649713 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:59.649790 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:59.650983 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.651673 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.652923 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.653359 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.653462 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:59.655841 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:59.655846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:59.655849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:59.655885 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:59.658686 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.659320 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.659324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.659326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.659452 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.659786 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.659790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.659792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.660069 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:59.660251 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:59.660563 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 20:23:59.660806 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 20:23:59.661767 16605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:59.745535 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 16474
I20260809 20:23:59.764883 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 16616
Aug 09 20:23:59 dist-test-slave-rnrw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:23:59.782451 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:59.765923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.12.78.2:34681)
0809 20:23:59.766593 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:59.766600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:59.766625 (+ 25us) server_negotiation.cc:407] Connection header received
0809 20:23:59.766685 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:59.766692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:59.766908 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:59.767092 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:59.768368 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.769489 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.770818 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.771395 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.771544 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:59.776357 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:59.776375 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:59.776396 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:59.776449 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:59.779399 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.780120 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.780129 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.780133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.780293 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.780746 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.780752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.780755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.781124 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:59.781444 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:59.781811 (+ 367us) server_negotiation.cc:299] Negotiation successful
0809 20:23:59.782098 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":365,"threads_started":1}
I20260809 20:23:59.783800 16745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:59.867367 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 16616
I20260809 20:23:59.886369 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 16755
Aug 09 20:23:59 dist-test-slave-rnrw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:23:59.904327 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:23:59.887382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.12.78.3:45193)
0809 20:23:59.888239 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0809 20:23:59.888246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:23:59.888278 (+ 32us) server_negotiation.cc:407] Connection header received
0809 20:23:59.888342 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:23:59.888349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:23:59.888682 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:23:59.888832 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:23:59.890120 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.891262 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.892514 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:23:59.893254 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:23:59.893360 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:23:59.897713 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:23:59.897731 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:23:59.897750 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:23:59.897793 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:23:59.901484 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.902170 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.902179 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.902183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.902327 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:23:59.902667 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:23:59.902672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:23:59.902675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:23:59.903124 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:23:59.903416 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0809 20:23:59.903645 (+ 229us) server_negotiation.cc:299] Negotiation successful
0809 20:23:59.903926 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":506,"threads_started":1}
I20260809 20:23:59.905768 16887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:23:59.989885 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 16755
I20260809 20:24:00.007727 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 16375
I20260809 20:24:00.009047 16441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:00.113840 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 16375
2026-08-09T20:24:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4014 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:24:00 dist-test-slave-rnrw krb5kdc[16912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:24:00 dist-test-slave-rnrw krb5kdc[16912](info): set up 2 sockets
Aug 09 20:24:00 dist-test-slave-rnrw krb5kdc[16912](info): commencing operation
krb5kdc: starting...
W20260809 20:24:01.261054 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:24:01 dist-test-slave-rnrw krb5kdc[16912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:24:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:24:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:24:01.501688 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:38087
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:38087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:01.846724 16926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:01.847110 16926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:01.847280 16926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:01.854732 16926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:01.854904 16926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:01.855016 16926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:01.855118 16926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:01.855206 16926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:24:01.865713 16926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:38087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:38087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:01.867166 16926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:01.868614 16926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:01.880834 16931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:01.880939 16932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:01.881484 16934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:01.881950 16926 server_base.cc:1034] running on GCE node
I20260809 20:24:01.882812 16926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:01.884337 16926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:01.885526 16926 hybrid_clock.cc:648] HybridClock initialized: now 1786307041885457 us; error 59 us; skew 500 ppm
I20260809 20:24:01.885942 16926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:01 dist-test-slave-rnrw krb5kdc[16912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307041, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:01.893996 16926 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:24:01.895390 16926 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:01.895498 16926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:01.895805 16926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:01.899166 16926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "ed9e1644bdfe497ba34ed50ec72e9c96"
format_stamp: "Formatted at 2026-08-09 20:24:01 on dist-test-slave-rnrw"
server_key: "b735a9b4de3762053d6da95f54c8158f"
server_key_iv: "161a097d7b296aa67e225e78dba66d91"
server_key_version: "encryptionkey@0"
I20260809 20:24:01.900027 16926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "ed9e1644bdfe497ba34ed50ec72e9c96"
format_stamp: "Formatted at 2026-08-09 20:24:01 on dist-test-slave-rnrw"
server_key: "b735a9b4de3762053d6da95f54c8158f"
server_key_iv: "161a097d7b296aa67e225e78dba66d91"
server_key_version: "encryptionkey@0"
I20260809 20:24:01.905838 16926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 20:24:01.909673 16940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:01.911171 16926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 20:24:01.911350 16926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "ed9e1644bdfe497ba34ed50ec72e9c96"
format_stamp: "Formatted at 2026-08-09 20:24:01 on dist-test-slave-rnrw"
server_key: "b735a9b4de3762053d6da95f54c8158f"
server_key_iv: "161a097d7b296aa67e225e78dba66d91"
server_key_version: "encryptionkey@0"
I20260809 20:24:01.911587 16926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:01.925709 16926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:01.930357 16926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:01.930616 16926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:01.950513 16926 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:38087
I20260809 20:24:01.950487 16992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:38087 every 8 connection(s)
I20260809 20:24:01.951502 16926 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:24:01.955711 16993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:01.955705 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 16926
I20260809 20:24:01.955921 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:24:01.956810 12600 external_mini_cluster.cc:1428] Setting key 9d1f839ef41d482f174783757ee23fa5
I20260809 20:24:01.970963 16993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96: Bootstrap starting.
Aug 09 20:24:01 dist-test-slave-rnrw krb5kdc[16912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:24:01.974905 16993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:01.976056 16993 log.cc:824] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:01.979099 16993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96: No bootstrap required, opened a new log
I20260809 20:24:01.981936 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:01.958966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.12.78.62:38087)
0809 20:24:01.959827 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:01.959838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:01.959880 (+ 42us) server_negotiation.cc:407] Connection header received
0809 20:24:01.962274 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:01.962297 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:01.962706 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:01.963047 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:01.965026 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:01.966853 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:01.968784 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:01.969479 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:01.969611 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:01.974376 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:01.974385 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:01.974401 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:01.974447 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:01.977641 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:01.978387 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:01.978393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:01.978400 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:01.978579 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:01.978971 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:01.978975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:01.978977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:01.979595 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:01.979934 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:01.980381 (+ 447us) server_negotiation.cc:299] Negotiation successful
0809 20:24:01.980864 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":414,"threads_started":1}
I20260809 20:24:01.987955 16993 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9e1644bdfe497ba34ed50ec72e9c96" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 38087 } }
I20260809 20:24:01.988384 16993 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:01.988504 16993 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed9e1644bdfe497ba34ed50ec72e9c96, State: Initialized, Role: FOLLOWER
I20260809 20:24:01.989192 16993 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9e1644bdfe497ba34ed50ec72e9c96" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 38087 } }
I20260809 20:24:01.989440 16993 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:24:01.989588 16993 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:24:01.989758 16993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:01.992292 16993 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9e1644bdfe497ba34ed50ec72e9c96" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 38087 } }
I20260809 20:24:01.992961 16993 leader_election.cc:302] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed9e1644bdfe497ba34ed50ec72e9c96; no voters:
I20260809 20:24:01.993588 16993 leader_election.cc:288] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:24:01.993749 16998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:01.994510 16998 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [term 1 LEADER]: Becoming Leader. State: Replica: ed9e1644bdfe497ba34ed50ec72e9c96, State: Running, Role: LEADER
I20260809 20:24:01.995093 16998 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9e1644bdfe497ba34ed50ec72e9c96" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 38087 } }
I20260809 20:24:01.995926 16993 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:01.998442 17000 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed9e1644bdfe497ba34ed50ec72e9c96. Latest consensus state: current_term: 1 leader_uuid: "ed9e1644bdfe497ba34ed50ec72e9c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9e1644bdfe497ba34ed50ec72e9c96" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 38087 } } }
I20260809 20:24:01.998818 17000 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:01.999269 16999 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed9e1644bdfe497ba34ed50ec72e9c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9e1644bdfe497ba34ed50ec72e9c96" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 38087 } } }
I20260809 20:24:01.999600 16999 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:02.003602 17006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:02.008128 17006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:02.017287 17006 catalog_manager.cc:1347] Generated new cluster ID: f370d7687f4c4840878d0332f94f2b5d
I20260809 20:24:02.017426 17006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:02.045186 17006 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:24:02.047163 17006 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:02.055802 17006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96: Generated new TSK 0
I20260809 20:24:02.056732 17006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:24:02.120430 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:38087
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:02.475162 17024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:02.475457 17024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:02.475559 17024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:02.483702 17024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:02.483852 17024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:02.484030 17024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:24:02.494297 17024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:38087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:02.496042 17024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:02.497586 17024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:02.510774 17032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:02.510818 17033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:02.510842 17035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:02.511922 17024 server_base.cc:1034] running on GCE node
I20260809 20:24:02.512600 17024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:02.513600 17024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:02.514837 17024 hybrid_clock.cc:648] HybridClock initialized: now 1786307042514788 us; error 46 us; skew 500 ppm
I20260809 20:24:02.515174 17024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:02 dist-test-slave-rnrw krb5kdc[16912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307042, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:02.518497 17024 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:24:02.519691 17024 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:02.519783 17024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:02.520047 17024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:02.523067 17024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "6806cdbe83e449c1816aa619901235d7"
format_stamp: "Formatted at 2026-08-09 20:24:02 on dist-test-slave-rnrw"
server_key: "fcf0820d2953fd4aab0df53445d68ab3"
server_key_iv: "18669879156adbbc1c70c73618efaef7"
server_key_version: "encryptionkey@0"
I20260809 20:24:02.523859 17024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "6806cdbe83e449c1816aa619901235d7"
format_stamp: "Formatted at 2026-08-09 20:24:02 on dist-test-slave-rnrw"
server_key: "fcf0820d2953fd4aab0df53445d68ab3"
server_key_iv: "18669879156adbbc1c70c73618efaef7"
server_key_version: "encryptionkey@0"
I20260809 20:24:02.529110 17024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260809 20:24:02.532755 17042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:02.534161 17024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:24:02.534341 17024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "6806cdbe83e449c1816aa619901235d7"
format_stamp: "Formatted at 2026-08-09 20:24:02 on dist-test-slave-rnrw"
server_key: "fcf0820d2953fd4aab0df53445d68ab3"
server_key_iv: "18669879156adbbc1c70c73618efaef7"
server_key_version: "encryptionkey@0"
I20260809 20:24:02.534564 17024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:02.550930 17024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:02.555161 17024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:02.555437 17024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:02.556444 17024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:02.557792 17024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:02.557907 17024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:02.558058 17024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:02.558148 17024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:02.591620 17024 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:35823
I20260809 20:24:02.591681 17164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:35823 every 8 connection(s)
I20260809 20:24:02.592514 17024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:24:02.594799 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 17024
I20260809 20:24:02.595017 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:24:02.595873 12600 external_mini_cluster.cc:1428] Setting key d6daa8270379d7608127df1e6ffca099
Aug 09 20:24:02 dist-test-slave-rnrw krb5kdc[16912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307042, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:24:02.617944 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:02.595140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:38017 (local address 127.12.78.62:38087)
0809 20:24:02.595667 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:02.595674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:02.596738 (+ 1064us) server_negotiation.cc:407] Connection header received
0809 20:24:02.599125 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:02.599130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:02.599206 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:02.599299 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:02.601562 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:02.602384 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:02.604940 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:02.605340 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:02.605437 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:02.609470 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:02.609481 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:02.609488 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:02.609541 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:02.613310 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:02.614093 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:02.614101 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:02.614107 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:02.614267 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:02.614744 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:02.614752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:02.614758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:02.615179 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:02.615340 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:02.616798 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0809 20:24:02.617230 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":244,"threads_started":1}
I20260809 20:24:02.618906 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:02.595621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:38087 (local address 127.12.78.1:38017)
0809 20:24:02.596499 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:02.596597 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:02.598824 (+ 2227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:02.599584 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:02.599599 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:02.600467 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:02.601362 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:02.601395 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:02.602569 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:02.602578 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:02.604790 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:02.604808 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:02.605576 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:02.605586 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:02.605910 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:02.606686 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:02.606708 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:02.609194 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:02.613467 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:02.613475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:02.613490 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:02.613950 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:02.614375 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:02.614378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:02.614381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:02.614585 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:02.615342 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:02.615354 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:02.616119 (+ 765us) client_negotiation.cc:769] Sending connection context
0809 20:24:02.616478 (+ 359us) client_negotiation.cc:240] Negotiation successful
0809 20:24:02.617207 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":357,"threads_started":1}
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
I20260809 20:24:02.627185 17165 heartbeater.cc:347] Connected to a master server at 127.12.78.62:38087
I20260809 20:24:02.627651 17165 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:02.628567 17165 heartbeater.cc:511] Master 127.12.78.62:38087 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
I20260809 20:24:02.631191 16957 ts_manager.cc:194] Registered new tserver with Master: 6806cdbe83e449c1816aa619901235d7 (127.12.78.1:35823)
I20260809 20:24:02.632793 16957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:38017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:24:02.668519 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:38087
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:03.016320 17172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:03.016714 17172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:03.016917 17172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:03.024619 17172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:03.024784 17172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:03.024960 17172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:24:03.036090 17172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:38087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:03.037580 17172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:03.039049 17172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:03.051846 17177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:03.051934 17178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:03.051891 17180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:03.053162 17172 server_base.cc:1034] running on GCE node
I20260809 20:24:03.053742 17172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:03.054662 17172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:03.055881 17172 hybrid_clock.cc:648] HybridClock initialized: now 1786307043055833 us; error 59 us; skew 500 ppm
I20260809 20:24:03.056206 17172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:03 dist-test-slave-rnrw krb5kdc[16912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307043, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:03.059387 17172 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:24:03.060635 17172 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:03.060734 17172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:03.061012 17172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:03.064056 17172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "4e605614a46e48c5aedbd4c761c489dc"
format_stamp: "Formatted at 2026-08-09 20:24:03 on dist-test-slave-rnrw"
server_key: "fb5295ceae701fd21aed16ccd6113949"
server_key_iv: "5315de57de6d397ade6fdbe94ef7387f"
server_key_version: "encryptionkey@0"
I20260809 20:24:03.064930 17172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "4e605614a46e48c5aedbd4c761c489dc"
format_stamp: "Formatted at 2026-08-09 20:24:03 on dist-test-slave-rnrw"
server_key: "fb5295ceae701fd21aed16ccd6113949"
server_key_iv: "5315de57de6d397ade6fdbe94ef7387f"
server_key_version: "encryptionkey@0"
I20260809 20:24:03.070250 17172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 20:24:03.073850 17186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:03.075204 17172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 20:24:03.075373 17172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "4e605614a46e48c5aedbd4c761c489dc"
format_stamp: "Formatted at 2026-08-09 20:24:03 on dist-test-slave-rnrw"
server_key: "fb5295ceae701fd21aed16ccd6113949"
server_key_iv: "5315de57de6d397ade6fdbe94ef7387f"
server_key_version: "encryptionkey@0"
I20260809 20:24:03.075603 17172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:03.109444 17172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:03.113663 17172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:03.113937 17172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:03.115025 17172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:03.116503 17172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:03.116665 17172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 20:24:03.116793 17172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:03.116886 17172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:03.150729 17172 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:36337
I20260809 20:24:03.150830 17302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:36337 every 8 connection(s)
I20260809 20:24:03.151607 17172 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:24:03.154719 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 17172
I20260809 20:24:03.154915 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:24:03.155519 12600 external_mini_cluster.cc:1428] Setting key d178bfe4845a35f830c73ce6fc3b1363
Aug 09 20:24:03 dist-test-slave-rnrw krb5kdc[16912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307043, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:24:03.176041 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.154232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:58439 (local address 127.12.78.62:38087)
0809 20:24:03.154759 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:03.154766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:03.155892 (+ 1126us) server_negotiation.cc:407] Connection header received
0809 20:24:03.158237 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:03.158241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:03.158302 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:03.158396 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:03.160638 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.161352 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.163936 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.164364 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.164458 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.168663 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:03.168675 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:03.168683 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:03.168731 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:03.172473 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:03.173324 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:03.173332 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:03.173339 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:03.173522 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:03.173928 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:03.173936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:03.173942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:03.174338 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:03.174501 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:03.175264 (+ 763us) server_negotiation.cc:299] Negotiation successful
0809 20:24:03.175527 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":260,"threads_started":1}
I20260809 20:24:03.176767 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.154753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:38087 (local address 127.12.78.2:58439)
0809 20:24:03.155681 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:03.155747 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:03.157934 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:03.158662 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:03.158675 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:03.159517 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:03.160387 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.160414 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.161518 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.161528 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.163766 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.163784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.164616 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.164625 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.164957 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.165840 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:03.165865 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:03.168342 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:03.172691 (+ 4349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:03.172699 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:03.172712 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:03.173197 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:03.173643 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:03.173646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:03.173648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:03.173821 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:03.174481 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:03.174487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:03.174820 (+ 333us) client_negotiation.cc:769] Sending connection context
0809 20:24:03.175138 (+ 318us) client_negotiation.cc:240] Negotiation successful
0809 20:24:03.175575 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":375,"threads_started":1}
I20260809 20:24:03.178360 17303 heartbeater.cc:347] Connected to a master server at 127.12.78.62:38087
I20260809 20:24:03.178683 17303 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
I20260809 20:24:03.179435 17303 heartbeater.cc:511] Master 127.12.78.62:38087 requested a full tablet report, sending...
I20260809 20:24:03.181005 16957 ts_manager.cc:194] Registered new tserver with Master: 4e605614a46e48c5aedbd4c761c489dc (127.12.78.2:36337)
I20260809 20:24:03.181944 16957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:58439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:24:03.208393 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:38087
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:03.562299 17309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:03.562628 17309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:03.562796 17309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:03.570765 17309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:03.570923 17309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:03.571120 17309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:24:03.581554 17309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:38087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:03.582937 17309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:03.584410 17309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:03.597855 17314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:03.598543 17315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:03.598958 17318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:03.599289 17309 server_base.cc:1034] running on GCE node
I20260809 20:24:03.599912 17309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:03.601011 17309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:03.602222 17309 hybrid_clock.cc:648] HybridClock initialized: now 1786307043602177 us; error 39 us; skew 500 ppm
I20260809 20:24:03.602553 17309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:03 dist-test-slave-rnrw krb5kdc[16912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307043, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:03.605901 17309 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:24:03.607090 17309 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:03.607218 17309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:03.607570 17309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:03.610671 17309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "79b81c11fe4a46228c7288e7a63cf4c8"
format_stamp: "Formatted at 2026-08-09 20:24:03 on dist-test-slave-rnrw"
server_key: "60eda58d66573c6a18afae42839aaaac"
server_key_iv: "d87e92b0d7c30b24ac39df98a7fc4fdc"
server_key_version: "encryptionkey@0"
I20260809 20:24:03.611521 17309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "79b81c11fe4a46228c7288e7a63cf4c8"
format_stamp: "Formatted at 2026-08-09 20:24:03 on dist-test-slave-rnrw"
server_key: "60eda58d66573c6a18afae42839aaaac"
server_key_iv: "d87e92b0d7c30b24ac39df98a7fc4fdc"
server_key_version: "encryptionkey@0"
I20260809 20:24:03.616994 17309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260809 20:24:03.620640 17324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:03.622023 17309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:24:03.622207 17309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "79b81c11fe4a46228c7288e7a63cf4c8"
format_stamp: "Formatted at 2026-08-09 20:24:03 on dist-test-slave-rnrw"
server_key: "60eda58d66573c6a18afae42839aaaac"
server_key_iv: "d87e92b0d7c30b24ac39df98a7fc4fdc"
server_key_version: "encryptionkey@0"
I20260809 20:24:03.622432 17309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:03.636850 17165 heartbeater.cc:503] Master 127.12.78.62:38087 was elected leader, sending a full tablet report...
I20260809 20:24:03.637611 17309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:03.642220 17309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:03.642513 17309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:03.643522 17309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:03.644982 17309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:03.645097 17309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:03.645226 17309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:03.645316 17309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:03.676781 17309 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:36923
I20260809 20:24:03.676882 17450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:36923 every 8 connection(s)
I20260809 20:24:03.677716 17309 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:24:03.681565 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 17309
I20260809 20:24:03.681777 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:24:03.682233 12600 external_mini_cluster.cc:1428] Setting key 4ac78fa74c7d164032858468a9b08086
Aug 09 20:24:03 dist-test-slave-rnrw krb5kdc[16912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307043, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:24:03.699249 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.680730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:37517 (local address 127.12.78.62:38087)
0809 20:24:03.681275 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:03.681284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:03.682301 (+ 1017us) server_negotiation.cc:407] Connection header received
0809 20:24:03.684588 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:03.684593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:03.684685 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:03.684798 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:03.686479 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.687271 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.688975 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.689423 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.689556 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.693165 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:03.693170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:03.693173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:03.693208 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:03.695783 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:03.696843 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:03.696849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:03.696857 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:03.696985 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:03.697431 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:03.697435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:03.697437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:03.697728 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:03.697885 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:03.698605 (+ 720us) server_negotiation.cc:299] Negotiation successful
0809 20:24:03.698845 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":231,"threads_started":1}
I20260809 20:24:03.700276 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.681266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:38087 (local address 127.12.78.3:37517)
0809 20:24:03.682104 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:03.682159 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:03.684279 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:03.685067 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:03.685078 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:03.685648 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:03.686302 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.686323 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.687442 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.687452 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.688823 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.688836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.689668 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.689677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.689902 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.690541 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:03.690568 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:03.692921 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:03.695922 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:03.695932 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:03.695950 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:03.696673 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:03.697099 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:03.697103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:03.697107 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:03.697305 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:03.697857 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:03.697865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:03.698228 (+ 363us) client_negotiation.cc:769] Sending connection context
0809 20:24:03.698499 (+ 271us) client_negotiation.cc:240] Negotiation successful
0809 20:24:03.699083 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":377,"threads_started":1}
I20260809 20:24:03.701776 17452 heartbeater.cc:347] Connected to a master server at 127.12.78.62:38087
I20260809 20:24:03.702105 17452 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:03.702749 17452 heartbeater.cc:511] Master 127.12.78.62:38087 requested a full tablet report, sending...
I20260809 20:24:03.704103 16957 ts_manager.cc:194] Registered new tserver with Master: 79b81c11fe4a46228c7288e7a63cf4c8 (127.12.78.3:36923)
I20260809 20:24:03.704959 16957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:37517
I20260809 20:24:03.709977 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:24:03.727576 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.714505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.12.78.62:38087)
0809 20:24:03.714661 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:03.714665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:03.715221 (+ 556us) server_negotiation.cc:407] Connection header received
0809 20:24:03.715423 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:03.715431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:03.715504 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:03.715606 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:03.717168 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.717900 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.719168 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.719573 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.719705 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.722124 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:03.722129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:03.722132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:03.722163 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:03.724760 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:03.725430 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:03.725438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:03.725442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:03.725571 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:03.726010 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:03.726017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:03.726021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:03.726379 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:03.726652 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:03.726990 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 20:24:03.727260 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 20:24:03.731415 16957 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:24:03.738265 16957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:24:03.773653 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.750934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.12.78.1:35823)
0809 20:24:03.763579 (+ 12645us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:03.763593 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:03.765029 (+ 1436us) server_negotiation.cc:407] Connection header received
0809 20:24:03.765217 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:03.765223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:03.765500 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:03.765685 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:03.767112 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.768609 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.771075 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.772385 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.772535 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.772815 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:03.773108 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 20:24:03.773278 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12345,"thread_start_us":12688,"threads_started":1}
I20260809 20:24:03.773734 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.750583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:35823 (local address 127.0.0.1:34414)
0809 20:24:03.764677 (+ 14094us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:03.764732 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:03.764962 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:03.766063 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:03.766074 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:03.766145 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:03.766880 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.766899 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.768788 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.768793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.770863 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.770877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.772633 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.772639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.772861 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.772910 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 20:24:03.773063 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 20:24:03.773272 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13770,"mutex_wait_us":81,"spinlock_wait_cycles":18048,"thread_start_us":13529,"threads_started":1}
I20260809 20:24:03.775514 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.751488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.12.78.2:36337)
0809 20:24:03.763737 (+ 12249us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:03.763751 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:03.764867 (+ 1116us) server_negotiation.cc:407] Connection header received
0809 20:24:03.765250 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:03.765259 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:03.765484 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:03.765683 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:03.768261 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.769843 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.772609 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.774090 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.774187 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.774388 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:03.774963 (+ 575us) server_negotiation.cc:299] Negotiation successful
0809 20:24:03.775145 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12001,"thread_start_us":12254,"threads_started":1}
I20260809 20:24:03.776437 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.750882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:36337 (local address 127.0.0.1:54238)
0809 20:24:03.764679 (+ 13797us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:03.764730 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:03.764989 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:03.766062 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:03.766078 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:03.766151 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:03.768075 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.768088 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.770003 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.770009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.772385 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.772400 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.774330 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.774336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.774738 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.774780 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 20:24:03.775824 (+ 1044us) client_negotiation.cc:240] Negotiation successful
0809 20:24:03.775979 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13586,"mutex_wait_us":4,"spinlock_wait_cycles":10752,"thread_start_us":13302,"threads_started":1}
I20260809 20:24:03.777074 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.765626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.12.78.3:36923)
0809 20:24:03.766884 (+ 1258us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:03.766890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:03.767640 (+ 750us) server_negotiation.cc:407] Connection header received
0809 20:24:03.767877 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:03.767886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:03.768141 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:03.768332 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:03.769298 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.770636 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.773353 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.775081 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.775221 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.775455 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:03.776667 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0809 20:24:03.776834 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1098,"thread_start_us":946,"threads_started":1}
I20260809 20:24:03.777391 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.765484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:36923 (local address 127.0.0.1:51222)
0809 20:24:03.767511 (+ 2027us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:03.767529 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:03.767664 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:03.768671 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:03.768675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:03.768700 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:03.769085 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.769094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.770913 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.770919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.773163 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.773177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.775589 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.775596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.775956 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.775996 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 20:24:03.776090 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 20:24:03.776253 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1894,"mutex_wait_us":24,"thread_start_us":1757,"threads_started":1}
I20260809 20:24:03.778294 17095 tablet_service.cc:1467] Processing CreateTablet for tablet 1286a2385b1f462f9fa262eeb9412a44 (DEFAULT_TABLE table=test-table [id=e4c162f6cd7541049938df2dc4eb5f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:03.779966 17095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1286a2385b1f462f9fa262eeb9412a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:03.779942 17236 tablet_service.cc:1467] Processing CreateTablet for tablet 1286a2385b1f462f9fa262eeb9412a44 (DEFAULT_TABLE table=test-table [id=e4c162f6cd7541049938df2dc4eb5f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:03.781352 17236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1286a2385b1f462f9fa262eeb9412a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:03.780905 17377 tablet_service.cc:1467] Processing CreateTablet for tablet 1286a2385b1f462f9fa262eeb9412a44 (DEFAULT_TABLE table=test-table [id=e4c162f6cd7541049938df2dc4eb5f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:03.782199 17377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1286a2385b1f462f9fa262eeb9412a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:03.792920 17472 tablet_bootstrap.cc:492] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7: Bootstrap starting.
I20260809 20:24:03.796144 17472 tablet_bootstrap.cc:654] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:03.796675 17473 tablet_bootstrap.cc:492] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc: Bootstrap starting.
I20260809 20:24:03.798689 17474 tablet_bootstrap.cc:492] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8: Bootstrap starting.
I20260809 20:24:03.799048 17472 log.cc:824] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:03.800344 17473 tablet_bootstrap.cc:654] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:03.801714 17473 log.cc:824] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:03.802086 17474 tablet_bootstrap.cc:654] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:03.802605 17472 tablet_bootstrap.cc:492] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7: No bootstrap required, opened a new log
I20260809 20:24:03.802879 17472 ts_tablet_manager.cc:1397] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260809 20:24:03.803369 17474 log.cc:824] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:03.805025 17473 tablet_bootstrap.cc:492] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc: No bootstrap required, opened a new log
I20260809 20:24:03.805317 17473 ts_tablet_manager.cc:1397] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 20:24:03.806006 17474 tablet_bootstrap.cc:492] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8: No bootstrap required, opened a new log
I20260809 20:24:03.806260 17474 ts_tablet_manager.cc:1397] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260809 20:24:03.813916 17474 raft_consensus.cc:348] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6806cdbe83e449c1816aa619901235d7" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35823 } } peers { permanent_uuid: "4e605614a46e48c5aedbd4c761c489dc" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 36337 } } peers { permanent_uuid: "79b81c11fe4a46228c7288e7a63cf4c8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 36923 } }
I20260809 20:24:03.813956 17472 raft_consensus.cc:348] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6806cdbe83e449c1816aa619901235d7" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35823 } } peers { permanent_uuid: "4e605614a46e48c5aedbd4c761c489dc" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 36337 } } peers { permanent_uuid: "79b81c11fe4a46228c7288e7a63cf4c8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 36923 } }
I20260809 20:24:03.814333 17474 raft_consensus.cc:374] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:03.814424 17472 raft_consensus.cc:374] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:03.814425 17474 raft_consensus.cc:729] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b81c11fe4a46228c7288e7a63cf4c8, State: Initialized, Role: FOLLOWER
I20260809 20:24:03.814527 17472 raft_consensus.cc:729] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6806cdbe83e449c1816aa619901235d7, State: Initialized, Role: FOLLOWER
I20260809 20:24:03.815160 17474 consensus_queue.cc:260] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6806cdbe83e449c1816aa619901235d7" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35823 } } peers { permanent_uuid: "4e605614a46e48c5aedbd4c761c489dc" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 36337 } } peers { permanent_uuid: "79b81c11fe4a46228c7288e7a63cf4c8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 36923 } }
I20260809 20:24:03.815228 17472 consensus_queue.cc:260] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6806cdbe83e449c1816aa619901235d7" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35823 } } peers { permanent_uuid: "4e605614a46e48c5aedbd4c761c489dc" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 36337 } } peers { permanent_uuid: "79b81c11fe4a46228c7288e7a63cf4c8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 36923 } }
I20260809 20:24:03.816468 17452 heartbeater.cc:503] Master 127.12.78.62:38087 was elected leader, sending a full tablet report...
I20260809 20:24:03.816187 17473 raft_consensus.cc:348] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6806cdbe83e449c1816aa619901235d7" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35823 } } peers { permanent_uuid: "4e605614a46e48c5aedbd4c761c489dc" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 36337 } } peers { permanent_uuid: "79b81c11fe4a46228c7288e7a63cf4c8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 36923 } }
I20260809 20:24:03.816776 17473 raft_consensus.cc:374] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:03.816903 17473 raft_consensus.cc:729] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e605614a46e48c5aedbd4c761c489dc, State: Initialized, Role: FOLLOWER
I20260809 20:24:03.817762 17473 consensus_queue.cc:260] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6806cdbe83e449c1816aa619901235d7" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35823 } } peers { permanent_uuid: "4e605614a46e48c5aedbd4c761c489dc" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 36337 } } peers { permanent_uuid: "79b81c11fe4a46228c7288e7a63cf4c8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 36923 } }
I20260809 20:24:03.821048 17474 ts_tablet_manager.cc:1428] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260809 20:24:03.822110 17303 heartbeater.cc:503] Master 127.12.78.62:38087 was elected leader, sending a full tablet report...
I20260809 20:24:03.823158 17473 ts_tablet_manager.cc:1428] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260809 20:24:03.824885 17472 ts_tablet_manager.cc:1428] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7: Time spent starting tablet: real 0.022s user 0.014s sys 0.009s
W20260809 20:24:03.844867 17166 tablet.cc:2369] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:24:03.903537 17304 tablet.cc:2369] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:24:03.907554 17480 raft_consensus.cc:482] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:24:03.907945 17480 raft_consensus.cc:504] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6806cdbe83e449c1816aa619901235d7" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35823 } } peers { permanent_uuid: "4e605614a46e48c5aedbd4c761c489dc" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 36337 } } peers { permanent_uuid: "79b81c11fe4a46228c7288e7a63cf4c8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 36923 } }
I20260809 20:24:03.909678 17480 leader_election.cc:288] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e605614a46e48c5aedbd4c761c489dc (127.12.78.2:36337), 79b81c11fe4a46228c7288e7a63cf4c8 (127.12.78.3:36923)
I20260809 20:24:03.917214 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.910175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:35511 (local address 127.12.78.2:36337)
0809 20:24:03.910420 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:03.910430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:03.910984 (+ 554us) server_negotiation.cc:407] Connection header received
0809 20:24:03.911081 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:03.911088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:03.911193 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:03.911313 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:03.912158 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.913156 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.915125 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.916243 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.916347 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.916456 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:03.916882 (+ 426us) server_negotiation.cc:299] Negotiation successful
0809 20:24:03.916979 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 20:24:03.917388 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.910105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:36337 (local address 127.12.78.1:35511)
0809 20:24:03.910740 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:03.910757 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:03.910895 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:03.911490 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:03.911497 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:03.911537 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:03.911994 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.912007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.913284 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.913289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.914980 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.914993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.916476 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.916483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.916720 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.916759 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 20:24:03.916860 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 20:24:03.917013 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":369,"threads_started":1}
I20260809 20:24:03.918187 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.911232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:33195 (local address 127.12.78.3:36923)
0809 20:24:03.911454 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:03.911461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:03.911724 (+ 263us) server_negotiation.cc:407] Connection header received
0809 20:24:03.911963 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:03.911976 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:03.912060 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:03.912160 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:03.912811 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.913824 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.915914 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.917070 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.917199 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.917298 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:03.917844 (+ 546us) server_negotiation.cc:299] Negotiation successful
0809 20:24:03.918007 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 20:24:03.918203 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.910938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:36923 (local address 127.12.78.1:33195)
0809 20:24:03.911605 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:03.911621 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:03.911790 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:03.912183 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:03.912187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:03.912213 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:03.912612 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.912624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.913975 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.913979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.915739 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:03.915756 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.917403 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.917414 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:03.917681 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.917715 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 20:24:03.917788 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 20:24:03.917872 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":342,"threads_started":1}
I20260809 20:24:03.918561 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1286a2385b1f462f9fa262eeb9412a44" candidate_uuid: "6806cdbe83e449c1816aa619901235d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e605614a46e48c5aedbd4c761c489dc" is_pre_election: true
I20260809 20:24:03.919178 17257 raft_consensus.cc:2454] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6806cdbe83e449c1816aa619901235d7 in term 0.
I20260809 20:24:03.919315 17401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1286a2385b1f462f9fa262eeb9412a44" candidate_uuid: "6806cdbe83e449c1816aa619901235d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b81c11fe4a46228c7288e7a63cf4c8" is_pre_election: true
I20260809 20:24:03.919833 17401 raft_consensus.cc:2454] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6806cdbe83e449c1816aa619901235d7 in term 0.
I20260809 20:24:03.919858 17045 leader_election.cc:302] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e605614a46e48c5aedbd4c761c489dc, 6806cdbe83e449c1816aa619901235d7; no voters:
I20260809 20:24:03.920300 17480 raft_consensus.cc:2781] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:24:03.920466 17480 raft_consensus.cc:482] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:24:03.920640 17480 raft_consensus.cc:3037] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:03.923105 17480 raft_consensus.cc:504] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6806cdbe83e449c1816aa619901235d7" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35823 } } peers { permanent_uuid: "4e605614a46e48c5aedbd4c761c489dc" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 36337 } } peers { permanent_uuid: "79b81c11fe4a46228c7288e7a63cf4c8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 36923 } }
I20260809 20:24:03.923698 17480 leader_election.cc:288] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [CANDIDATE]: Term 1 election: Requested vote from peers 4e605614a46e48c5aedbd4c761c489dc (127.12.78.2:36337), 79b81c11fe4a46228c7288e7a63cf4c8 (127.12.78.3:36923)
I20260809 20:24:03.924168 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1286a2385b1f462f9fa262eeb9412a44" candidate_uuid: "6806cdbe83e449c1816aa619901235d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e605614a46e48c5aedbd4c761c489dc"
I20260809 20:24:03.924256 17401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1286a2385b1f462f9fa262eeb9412a44" candidate_uuid: "6806cdbe83e449c1816aa619901235d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b81c11fe4a46228c7288e7a63cf4c8"
I20260809 20:24:03.924378 17257 raft_consensus.cc:3037] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:03.924503 17401 raft_consensus.cc:3037] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:03.926882 17257 raft_consensus.cc:2454] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6806cdbe83e449c1816aa619901235d7 in term 1.
I20260809 20:24:03.926935 17401 raft_consensus.cc:2454] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6806cdbe83e449c1816aa619901235d7 in term 1.
I20260809 20:24:03.927299 17045 leader_election.cc:302] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e605614a46e48c5aedbd4c761c489dc, 6806cdbe83e449c1816aa619901235d7; no voters:
I20260809 20:24:03.927640 17480 raft_consensus.cc:2781] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:03.928102 17480 raft_consensus.cc:686] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [term 1 LEADER]: Becoming Leader. State: Replica: 6806cdbe83e449c1816aa619901235d7, State: Running, Role: LEADER
I20260809 20:24:03.928907 17480 consensus_queue.cc:237] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6806cdbe83e449c1816aa619901235d7" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35823 } } peers { permanent_uuid: "4e605614a46e48c5aedbd4c761c489dc" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 36337 } } peers { permanent_uuid: "79b81c11fe4a46228c7288e7a63cf4c8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 36923 } }
W20260809 20:24:03.929203 17453 tablet.cc:2369] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:24:03.933413 16956 catalog_manager.cc:5495] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6806cdbe83e449c1816aa619901235d7 (127.12.78.1). New cstate: current_term: 1 leader_uuid: "6806cdbe83e449c1816aa619901235d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6806cdbe83e449c1816aa619901235d7" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e605614a46e48c5aedbd4c761c489dc" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 36337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79b81c11fe4a46228c7288e7a63cf4c8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 36923 } health_report { overall_health: UNKNOWN } } }
I20260809 20:24:03.971721 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:03.966437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.12.78.1:35823)
0809 20:24:03.966632 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:03.966637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:03.966850 (+ 213us) server_negotiation.cc:407] Connection header received
0809 20:24:03.967037 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:03.967041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:03.967138 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:03.967265 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:24:03.967642 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.968342 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.969700 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:03.970104 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:03.970206 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:03.970697 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:24:03.971223 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:24:03.971330 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:03.971447 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 20:24:03.971515 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 20:24:03.980789 17257 raft_consensus.cc:1261] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc [term 1 FOLLOWER]: Refusing update from remote peer 6806cdbe83e449c1816aa619901235d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:03.980789 17401 raft_consensus.cc:1261] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8 [term 1 FOLLOWER]: Refusing update from remote peer 6806cdbe83e449c1816aa619901235d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:03.981979 17484 consensus_queue.cc:1035] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e605614a46e48c5aedbd4c761c489dc" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 36337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:03.982445 17480 consensus_queue.cc:1035] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b81c11fe4a46228c7288e7a63cf4c8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 36923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:03.995152 17487 mvcc.cc:205] Tried to move back new op lower bound from 7316713652132614144 to 7316713651936665600. Current Snapshot: MvccSnapshot[applied={T|T < 7316713652132614144}]
I20260809 20:24:03.999009 17489 mvcc.cc:205] Tried to move back new op lower bound from 7316713652132614144 to 7316713651936665600. Current Snapshot: MvccSnapshot[applied={T|T < 7316713652132614144}]
I20260809 20:24:04.009572 16956 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:04.009912 16956 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:04.014281 16956 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed9e1644bdfe497ba34ed50ec72e9c96: Sending DeleteTablet for 3 replicas of tablet 1286a2385b1f462f9fa262eeb9412a44
I20260809 20:24:04.015652 17095 tablet_service.cc:1514] Processing DeleteTablet for tablet 1286a2385b1f462f9fa262eeb9412a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:04 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:34414
I20260809 20:24:04.015790 17377 tablet_service.cc:1514] Processing DeleteTablet for tablet 1286a2385b1f462f9fa262eeb9412a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:04 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:51222
I20260809 20:24:04.015889 17236 tablet_service.cc:1514] Processing DeleteTablet for tablet 1286a2385b1f462f9fa262eeb9412a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:04 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:54238
I20260809 20:24:04.018477 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 17024
I20260809 20:24:04.027257 17497 tablet_replica.cc:330] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7: stopping tablet replica
I20260809 20:24:04.028090 17497 raft_consensus.cc:2229] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:24:04.029151 17499 tablet_replica.cc:330] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc: stopping tablet replica
I20260809 20:24:04.029837 17497 raft_consensus.cc:2258] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:04.030073 17498 tablet_replica.cc:330] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8: stopping tablet replica
I20260809 20:24:04.030532 17498 raft_consensus.cc:2229] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:04.030791 17499 raft_consensus.cc:2229] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:04.031072 17498 raft_consensus.cc:2258] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:04.033115 17497 ts_tablet_manager.cc:1905] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:04.033118 17498 ts_tablet_manager.cc:1905] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:04.033701 17499 raft_consensus.cc:2258] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:04.036052 17499 ts_tablet_manager.cc:1905] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 20:24:04 dist-test-slave-rnrw krb5kdc[16912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:24:04.040619 17498 ts_tablet_manager.cc:1918] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:04.040814 17498 log.cc:1196] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/1286a2385b1f462f9fa262eeb9412a44
I20260809 20:24:04.041424 17498 ts_tablet_manager.cc:1939] T 1286a2385b1f462f9fa262eeb9412a44 P 79b81c11fe4a46228c7288e7a63cf4c8: Deleting consensus metadata
I20260809 20:24:04.044474 16943 catalog_manager.cc:4841] TS 79b81c11fe4a46228c7288e7a63cf4c8 (127.12.78.3:36923): tablet 1286a2385b1f462f9fa262eeb9412a44 (table test-table [id=e4c162f6cd7541049938df2dc4eb5f45]) successfully deleted
I20260809 20:24:04.048133 17499 ts_tablet_manager.cc:1918] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:04.048334 17499 log.cc:1196] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/1286a2385b1f462f9fa262eeb9412a44
I20260809 20:24:04.048979 17499 ts_tablet_manager.cc:1939] T 1286a2385b1f462f9fa262eeb9412a44 P 4e605614a46e48c5aedbd4c761c489dc: Deleting consensus metadata
I20260809 20:24:04.051015 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:04.020300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.12.78.1:35823)
0809 20:24:04.020627 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:04.020637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:04.029392 (+ 8755us) server_negotiation.cc:407] Connection header received
0809 20:24:04.029615 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:04.029620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:04.029718 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:04.029843 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:04.032115 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:04.033284 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:04.035208 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:04.035902 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:04.036003 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:04.041908 (+ 5905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:04.041925 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:04.041951 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:04.042005 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:04.046881 (+ 4876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:04.047890 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:04.047906 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:04.047910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:04.048122 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:04.048674 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:04.048683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:04.048686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:04.049264 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:04.049637 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:04.050260 (+ 623us) server_negotiation.cc:299] Negotiation successful
0809 20:24:04.050543 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 20:24:04.051038 16943 catalog_manager.cc:4841] TS 4e605614a46e48c5aedbd4c761c489dc (127.12.78.2:36337): tablet 1286a2385b1f462f9fa262eeb9412a44 (table test-table [id=e4c162f6cd7541049938df2dc4eb5f45]) successfully deleted
I20260809 20:24:04.052695 17159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:04.151331 17497 ts_tablet_manager.cc:1918] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:04.151618 17497 log.cc:1196] T 1286a2385b1f462f9fa262eeb9412a44 P 6806cdbe83e449c1816aa619901235d7: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/1286a2385b1f462f9fa262eeb9412a44
I20260809 20:24:04.152052 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 17024
W20260809 20:24:04.172356 16941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.78.1:35823 (error 108)
I20260809 20:24:04.172699 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 17172
I20260809 20:24:04.174477 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:04.174019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:35823 (local address 127.0.0.1:34432)
0809 20:24:04.174162 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:04.174302 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:35823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 20:24:04.174749 16941 catalog_manager.cc:4584] TS 6806cdbe83e449c1816aa619901235d7 (127.12.78.1:35823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1286a2385b1f462f9fa262eeb9412a44: Network error: Client connection negotiation failed: client connection to 127.12.78.1:35823: connect: Connection refused (error 111)
Aug 09 20:24:04 dist-test-slave-rnrw krb5kdc[16912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:24:04.189143 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:04.173668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.12.78.2:36337)
0809 20:24:04.173859 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:04.173865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:04.173888 (+ 23us) server_negotiation.cc:407] Connection header received
0809 20:24:04.174067 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:04.174072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:04.174159 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:04.174270 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:04.175702 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:04.176460 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:04.177809 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:04.178196 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:04.178291 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:04.183067 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:04.183082 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:04.183098 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:04.183139 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:04.186423 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:04.187121 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:04.187136 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:04.187138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:04.187280 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:04.187627 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:04.187631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:04.187633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:04.188086 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:04.188323 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:04.188686 (+ 363us) server_negotiation.cc:299] Negotiation successful
0809 20:24:04.188881 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 20:24:04.189894 17298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:04.228710 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:04.228147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:35823 (local address 127.0.0.1:34448)
0809 20:24:04.228360 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:04.228478 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:35823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 20:24:04.229825 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:04.229423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:35823 (local address 127.0.0.1:34464)
0809 20:24:04.229585 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:04.229690 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:35823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 20:24:04.283572 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 17172
I20260809 20:24:04.290429 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:04.289943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:35823 (local address 127.0.0.1:34480)
0809 20:24:04.290169 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:04.290271 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:35823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260809 20:24:04.291591 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:04.291266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:35823 (local address 127.0.0.1:34484)
0809 20:24:04.291407 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:04.291478 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:35823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 20:24:04.305860 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 17309
Aug 09 20:24:04 dist-test-slave-rnrw krb5kdc[16912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:24:04.322852 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:04.306730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.12.78.3:36923)
0809 20:24:04.306932 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:04.306941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:04.306964 (+ 23us) server_negotiation.cc:407] Connection header received
0809 20:24:04.307197 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:04.307202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:04.307275 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:04.307392 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:04.308925 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:04.309746 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:04.311107 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:04.311552 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:04.311718 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:04.316633 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:04.316652 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:04.316670 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:04.316721 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:04.319668 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:04.320439 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:04.320454 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:04.320456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:04.320675 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:04.321082 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:04.321088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:04.321092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:04.321504 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:04.321860 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:04.322211 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 20:24:04.322477 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 20:24:04.323540 17446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:04.372601 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:04.371979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:35823 (local address 127.0.0.1:34492)
0809 20:24:04.372216 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:04.372319 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:35823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260809 20:24:04.373828 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:04.373363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:35823 (local address 127.0.0.1:34502)
0809 20:24:04.373643 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:04.373716 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:35823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260809 20:24:04.413519 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 17309
I20260809 20:24:04.432412 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 16926
I20260809 20:24:04.433734 16988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:04.544888 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 16926
2026-08-09T20:24:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4431 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T20:24:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:24:04Z Disabled control of system clock
I20260809 20:24:04.619499 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44219
--webserver_interface=127.12.78.62
--webserver_port=0
--builtin_ntp_servers=127.12.78.20:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:44219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 20:24:04.959700 17509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:04.959996 17509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:04.960155 17509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:04.967646 17509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:04.967778 17509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:04.967854 17509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:04.967940 17509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:04.968015 17509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:24:04.977608 17509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:44219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:04.979053 17509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:04.980999 17509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:04.993861 17518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:04.993952 17515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:04.994145 17516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:04.995010 17509 server_base.cc:1034] running on GCE node
I20260809 20:24:04.996081 17509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:04.997931 17509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:04.999221 17509 hybrid_clock.cc:648] HybridClock initialized: now 1786307044999181 us; error 60 us; skew 500 ppm
I20260809 20:24:04.999691 17509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:05.002467 17509 webserver.cc:460] Webserver started at http://127.12.78.62:38851/ using document root <none> and password file <none>
I20260809 20:24:05.003527 17509 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:05.003666 17509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:05.004029 17509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:05.007329 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "4c14b763849644488f716b487d5c4b39"
format_stamp: "Formatted at 2026-08-09 20:24:04 on dist-test-slave-rnrw"
server_key: "5586926a49759cec1e139db2e25e16d9"
server_key_iv: "f4e91ed4e0ae33e967badc034310aba1"
server_key_version: "encryptionkey@0"
I20260809 20:24:05.008289 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "4c14b763849644488f716b487d5c4b39"
format_stamp: "Formatted at 2026-08-09 20:24:04 on dist-test-slave-rnrw"
server_key: "5586926a49759cec1e139db2e25e16d9"
server_key_iv: "f4e91ed4e0ae33e967badc034310aba1"
server_key_version: "encryptionkey@0"
I20260809 20:24:05.014045 17509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 20:24:05.017755 17525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:05.019302 17509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 20:24:05.019500 17509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "4c14b763849644488f716b487d5c4b39"
format_stamp: "Formatted at 2026-08-09 20:24:04 on dist-test-slave-rnrw"
server_key: "5586926a49759cec1e139db2e25e16d9"
server_key_iv: "f4e91ed4e0ae33e967badc034310aba1"
server_key_version: "encryptionkey@0"
I20260809 20:24:05.019743 17509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:05.025820 17509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:05.026654 17509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:05.027050 17509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:05.046113 17509 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:44219
I20260809 20:24:05.046165 17580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:44219 every 8 connection(s)
I20260809 20:24:05.047542 17509 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:24:05.050140 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 17509
I20260809 20:24:05.050379 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:24:05.051209 12600 external_mini_cluster.cc:1428] Setting key 7facb840635fb6c63439b798c8743cf3
I20260809 20:24:05.052289 17581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:05.060497 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:05.053362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.12.78.62:44219)
0809 20:24:05.054365 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:05.054375 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:05.054416 (+ 41us) server_negotiation.cc:407] Connection header received
0809 20:24:05.056857 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:05.056878 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:05.057315 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:05.057663 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:05.058886 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:05.058893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:05.058908 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:05.058924 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:05.058989 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:05.059005 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:05.059188 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:05.059290 (+ 102us) server_negotiation.cc:299] Negotiation successful
0809 20:24:05.059605 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":402,"threads_started":1}
I20260809 20:24:05.065347 17581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39: Bootstrap starting.
I20260809 20:24:05.068233 17581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:05.069322 17581 log.cc:824] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:05.072046 17581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39: No bootstrap required, opened a new log
I20260809 20:24:05.079382 17581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c14b763849644488f716b487d5c4b39" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44219 } }
I20260809 20:24:05.079811 17581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:05.079926 17581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c14b763849644488f716b487d5c4b39, State: Initialized, Role: FOLLOWER
I20260809 20:24:05.080502 17581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c14b763849644488f716b487d5c4b39" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44219 } }
I20260809 20:24:05.080791 17581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:24:05.080896 17581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:24:05.081017 17581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:05.083135 17581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c14b763849644488f716b487d5c4b39" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44219 } }
I20260809 20:24:05.083588 17581 leader_election.cc:302] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c14b763849644488f716b487d5c4b39; no voters:
I20260809 20:24:05.084153 17581 leader_election.cc:288] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:24:05.084306 17586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:05.085083 17586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [term 1 LEADER]: Becoming Leader. State: Replica: 4c14b763849644488f716b487d5c4b39, State: Running, Role: LEADER
I20260809 20:24:05.085616 17586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c14b763849644488f716b487d5c4b39" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44219 } }
I20260809 20:24:05.086504 17581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:05.089283 17587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c14b763849644488f716b487d5c4b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c14b763849644488f716b487d5c4b39" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44219 } } }
I20260809 20:24:05.089716 17587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:05.091605 17588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c14b763849644488f716b487d5c4b39. Latest consensus state: current_term: 1 leader_uuid: "4c14b763849644488f716b487d5c4b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c14b763849644488f716b487d5c4b39" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44219 } } }
I20260809 20:24:05.092236 17588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:05.093101 17596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:05.098325 17596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:05.108283 17596 catalog_manager.cc:1347] Generated new cluster ID: 2d17fad98ab74cc39a562f4d7da25486
I20260809 20:24:05.108428 17596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:05.128036 17596 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:24:05.128508 17596 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:05.135560 17596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39: Generated new TSK 0
I20260809 20:24:05.136384 17596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:24:05.149183 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44219
--builtin_ntp_servers=127.12.78.20:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 20:24:05.503644 17607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:05.503938 17607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:05.504096 17607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:05.512001 17607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:05.512118 17607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:05.512291 17607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:24:05.521898 17607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:05.523298 17607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:05.525236 17607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:05.538558 17613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:05.538653 17615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:05.538532 17612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:05.539422 17607 server_base.cc:1034] running on GCE node
I20260809 20:24:05.540011 17607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:05.541025 17607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:05.542263 17607 hybrid_clock.cc:648] HybridClock initialized: now 1786307045542217 us; error 52 us; skew 500 ppm
I20260809 20:24:05.542629 17607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:05.545068 17607 webserver.cc:460] Webserver started at http://127.12.78.1:35739/ using document root <none> and password file <none>
I20260809 20:24:05.545781 17607 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:05.545909 17607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:05.546249 17607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:05.549223 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "2e552729fb5c47f18b0870d8287bea7b"
format_stamp: "Formatted at 2026-08-09 20:24:05 on dist-test-slave-rnrw"
server_key: "b29225cfe1a9afb32ad9a6f5b8516310"
server_key_iv: "3bbf79a9fa1ead0f09e9dd64ed855264"
server_key_version: "encryptionkey@0"
I20260809 20:24:05.550027 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "2e552729fb5c47f18b0870d8287bea7b"
format_stamp: "Formatted at 2026-08-09 20:24:05 on dist-test-slave-rnrw"
server_key: "b29225cfe1a9afb32ad9a6f5b8516310"
server_key_iv: "3bbf79a9fa1ead0f09e9dd64ed855264"
server_key_version: "encryptionkey@0"
I20260809 20:24:05.555187 17607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:24:05.558708 17622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:05.560068 17607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 20:24:05.560250 17607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "2e552729fb5c47f18b0870d8287bea7b"
format_stamp: "Formatted at 2026-08-09 20:24:05 on dist-test-slave-rnrw"
server_key: "b29225cfe1a9afb32ad9a6f5b8516310"
server_key_iv: "3bbf79a9fa1ead0f09e9dd64ed855264"
server_key_version: "encryptionkey@0"
I20260809 20:24:05.560479 17607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:05.566566 17607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:05.567637 17607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:05.567938 17607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:05.568961 17607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:05.570428 17607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:05.570559 17607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:05.570693 17607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:05.570777 17607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:05.604712 17607 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:38099
I20260809 20:24:05.604759 17743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:38099 every 8 connection(s)
I20260809 20:24:05.606091 17607 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:24:05.613088 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 17607
I20260809 20:24:05.613334 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:24:05.614151 12600 external_mini_cluster.cc:1428] Setting key 98b80fe5cb83859900f38cdf927b493a
I20260809 20:24:05.614610 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:05.608689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:47581 (local address 127.12.78.62:44219)
0809 20:24:05.609176 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:05.609180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:05.610263 (+ 1083us) server_negotiation.cc:407] Connection header received
0809 20:24:05.612018 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:05.612024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:05.612099 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:05.612226 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:05.613844 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:05.613848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:05.613852 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:05.613869 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:05.613914 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:05.613925 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:05.614032 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:05.614295 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 20:24:05.614366 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":263,"threads_started":1}
I20260809 20:24:05.615864 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:05.609183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44219 (local address 127.12.78.1:47581)
0809 20:24:05.610099 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:05.610142 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:05.611733 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:05.612488 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:05.612498 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:05.612652 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:05.613442 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:05.613446 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:05.613499 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:05.613640 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:05.614031 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:05.614044 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:05.614052 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 20:24:05.614374 (+ 322us) client_negotiation.cc:240] Negotiation successful
0809 20:24:05.614681 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":282,"threads_started":1}
I20260809 20:24:05.617056 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44219
--builtin_ntp_servers=127.12.78.20:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 20:24:05.617357 17744 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44219
I20260809 20:24:05.617681 17744 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:05.618412 17744 heartbeater.cc:511] Master 127.12.78.62:44219 requested a full tablet report, sending...
I20260809 20:24:05.620671 17543 ts_manager.cc:194] Registered new tserver with Master: 2e552729fb5c47f18b0870d8287bea7b (127.12.78.1:38099)
W20260809 20:24:05.954473 17748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:05.954746 17748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:05.954834 17748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:05.961993 17748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:05.962090 17748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:05.962198 17748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:24:05.971212 17748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:05.972447 17748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:05.974263 17748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:05.987046 17754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:05.987174 17757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:05.987183 17755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:05.987793 17748 server_base.cc:1034] running on GCE node
I20260809 20:24:05.988642 17748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:05.989797 17748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:05.991055 17748 hybrid_clock.cc:648] HybridClock initialized: now 1786307045990985 us; error 54 us; skew 500 ppm
I20260809 20:24:05.991406 17748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:05.993786 17748 webserver.cc:460] Webserver started at http://127.12.78.2:43277/ using document root <none> and password file <none>
I20260809 20:24:05.994518 17748 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:05.994652 17748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:05.994997 17748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:05.997998 17748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "59d8652f66344301830c5c76d83193cf"
format_stamp: "Formatted at 2026-08-09 20:24:05 on dist-test-slave-rnrw"
server_key: "f744e96d139c321714664538861618ce"
server_key_iv: "e3d7015a76b5384ae807f8f0beda40f8"
server_key_version: "encryptionkey@0"
I20260809 20:24:05.998813 17748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "59d8652f66344301830c5c76d83193cf"
format_stamp: "Formatted at 2026-08-09 20:24:05 on dist-test-slave-rnrw"
server_key: "f744e96d139c321714664538861618ce"
server_key_iv: "e3d7015a76b5384ae807f8f0beda40f8"
server_key_version: "encryptionkey@0"
I20260809 20:24:06.004165 17748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 20:24:06.007736 17763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:06.009182 17748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:06.009362 17748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "59d8652f66344301830c5c76d83193cf"
format_stamp: "Formatted at 2026-08-09 20:24:05 on dist-test-slave-rnrw"
server_key: "f744e96d139c321714664538861618ce"
server_key_iv: "e3d7015a76b5384ae807f8f0beda40f8"
server_key_version: "encryptionkey@0"
I20260809 20:24:06.009601 17748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:06.015278 17748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:06.016057 17748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:06.016333 17748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:06.017313 17748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:06.018694 17748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:06.018814 17748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:06.018939 17748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:06.019023 17748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:06.052639 17748 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:45059
I20260809 20:24:06.052696 17880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:45059 every 8 connection(s)
I20260809 20:24:06.053937 17748 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:24:06.060153 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 17748
I20260809 20:24:06.060349 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:24:06.060907 12600 external_mini_cluster.cc:1428] Setting key dd6ec34739b6183d3e4c6f12ac3c32e4
I20260809 20:24:06.063951 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.056602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:53251 (local address 127.12.78.62:44219)
0809 20:24:06.056821 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:06.056827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:06.058403 (+ 1576us) server_negotiation.cc:407] Connection header received
0809 20:24:06.060825 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:06.060831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:06.060901 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:06.061024 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:06.063132 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:06.063138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:06.063143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:06.063161 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:06.063198 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:06.063214 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:06.063334 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:06.063538 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 20:24:06.063606 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 20:24:06.064018 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44219
--builtin_ntp_servers=127.12.78.20:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 20:24:06.064967 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.057163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44219 (local address 127.12.78.2:53251)
0809 20:24:06.058239 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:06.058282 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:06.060483 (+ 2201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:06.061296 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:06.061308 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:06.061432 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:06.062756 (+ 1324us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:06.062759 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:06.062822 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:06.062926 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:06.063319 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:06.063328 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:06.063336 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 20:24:06.063603 (+ 267us) client_negotiation.cc:240] Negotiation successful
0809 20:24:06.063853 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":306,"threads_started":1}
I20260809 20:24:06.066313 17881 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44219
I20260809 20:24:06.066641 17881 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:06.067344 17881 heartbeater.cc:511] Master 127.12.78.62:44219 requested a full tablet report, sending...
I20260809 20:24:06.068913 17543 ts_manager.cc:194] Registered new tserver with Master: 59d8652f66344301830c5c76d83193cf (127.12.78.2:45059)
W20260809 20:24:06.405575 17884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:06.405858 17884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:06.406023 17884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:06.413399 17884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:06.413524 17884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:06.413709 17884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:24:06.422953 17884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:06.424402 17884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:06.426394 17884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:06.440284 17889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:06.440196 17892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:06.440254 17890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:06.441197 17884 server_base.cc:1034] running on GCE node
I20260809 20:24:06.441828 17884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:06.442804 17884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:06.444041 17884 hybrid_clock.cc:648] HybridClock initialized: now 1786307046444015 us; error 68 us; skew 500 ppm
I20260809 20:24:06.444368 17884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:06.446822 17884 webserver.cc:460] Webserver started at http://127.12.78.3:46287/ using document root <none> and password file <none>
I20260809 20:24:06.447547 17884 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:06.447707 17884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:06.448053 17884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:06.450992 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "e0da79877ccc4fb088081dece170111b"
format_stamp: "Formatted at 2026-08-09 20:24:06 on dist-test-slave-rnrw"
server_key: "ed60be260bbd7b19ca77da0f0cdaa94b"
server_key_iv: "e977d4577f51128ee76885c211d32467"
server_key_version: "encryptionkey@0"
I20260809 20:24:06.451802 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "e0da79877ccc4fb088081dece170111b"
format_stamp: "Formatted at 2026-08-09 20:24:06 on dist-test-slave-rnrw"
server_key: "ed60be260bbd7b19ca77da0f0cdaa94b"
server_key_iv: "e977d4577f51128ee76885c211d32467"
server_key_version: "encryptionkey@0"
I20260809 20:24:06.456974 17884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 20:24:06.460297 17898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:06.461655 17884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:06.461822 17884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "e0da79877ccc4fb088081dece170111b"
format_stamp: "Formatted at 2026-08-09 20:24:06 on dist-test-slave-rnrw"
server_key: "ed60be260bbd7b19ca77da0f0cdaa94b"
server_key_iv: "e977d4577f51128ee76885c211d32467"
server_key_version: "encryptionkey@0"
I20260809 20:24:06.462036 17884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:06.468123 17884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:06.468982 17884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:06.469249 17884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:06.470125 17884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:06.471506 17884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:06.471630 17884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:06.471747 17884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:06.471832 17884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:06.506539 17884 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:35837
I20260809 20:24:06.506628 18016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:35837 every 8 connection(s)
I20260809 20:24:06.507984 17884 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:24:06.516215 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.510525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:46263 (local address 127.12.78.62:44219)
0809 20:24:06.510788 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:06.510795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:06.512230 (+ 1435us) server_negotiation.cc:407] Connection header received
0809 20:24:06.514075 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:06.514079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:06.514136 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:06.514226 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:06.515434 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:06.515439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:06.515443 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:06.515462 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:06.515498 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:06.515510 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:06.515607 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:06.515934 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 20:24:06.515993 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 20:24:06.517024 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.511118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44219 (local address 127.12.78.3:46263)
0809 20:24:06.512063 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:06.512110 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:06.513761 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:06.514420 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:06.514428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:06.514515 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:06.515148 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:06.515150 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:06.515187 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:06.515268 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:06.515594 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:06.515602 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:06.515607 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 20:24:06.515829 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 20:24:06.516019 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":291,"threads_started":1}
I20260809 20:24:06.518023 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 17884
I20260809 20:24:06.518224 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:24:06.518585 18017 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44219
I20260809 20:24:06.518859 12600 external_mini_cluster.cc:1428] Setting key c74a940c21975133e05df02526f08361
I20260809 20:24:06.518920 18017 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:06.519621 18017 heartbeater.cc:511] Master 127.12.78.62:44219 requested a full tablet report, sending...
I20260809 20:24:06.521159 17543 ts_manager.cc:194] Registered new tserver with Master: e0da79877ccc4fb088081dece170111b (127.12.78.3:35837)
I20260809 20:24:06.522243 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:24:06.529733 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.527265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.12.78.62:44219)
0809 20:24:06.527420 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:06.527425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:06.527973 (+ 548us) server_negotiation.cc:407] Connection header received
0809 20:24:06.528097 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:06.528100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:06.528153 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:06.528240 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:06.529229 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:06.529233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:06.529237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:06.529251 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:06.529283 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:06.529295 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:06.529394 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:06.529480 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 20:24:06.529534 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 20:24:06.533790 17541 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:24:06.541651 17541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:24:06.563890 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.553930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:45059 (local address 127.0.0.1:36606)
0809 20:24:06.556494 (+ 2564us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:06.556514 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:06.556667 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:06.560133 (+ 3466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:06.560144 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:06.560263 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:06.561210 (+ 947us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:06.561216 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:06.561257 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:06.561905 (+ 648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:06.562512 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:06.562524 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:06.562533 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 20:24:06.562751 (+ 218us) client_negotiation.cc:240] Negotiation successful
0809 20:24:06.563395 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2379,"thread_start_us":2471,"threads_started":1}
I20260809 20:24:06.564051 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.554502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.12.78.2:45059)
0809 20:24:06.555934 (+ 1432us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:06.555946 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:06.556816 (+ 870us) server_negotiation.cc:407] Connection header received
0809 20:24:06.556911 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:06.556923 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:06.557225 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:06.557377 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:06.562113 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:06.562123 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:06.562140 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:06.562176 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:06.562245 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:06.562270 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:06.562454 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:06.563202 (+ 748us) server_negotiation.cc:299] Negotiation successful
0809 20:24:06.563578 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1235,"thread_start_us":1666,"threads_started":1}
I20260809 20:24:06.564880 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.553930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.12.78.3:35837)
0809 20:24:06.561903 (+ 7973us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:06.561912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:06.561949 (+ 37us) server_negotiation.cc:407] Connection header received
0809 20:24:06.562018 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:06.562027 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:06.562268 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:06.562420 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:06.563646 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:06.563657 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:06.563669 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:06.563687 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:06.563744 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:06.563766 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:06.563946 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:06.564409 (+ 463us) server_negotiation.cc:299] Negotiation successful
0809 20:24:06.564500 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7807,"thread_start_us":6984,"threads_started":1}
I20260809 20:24:06.565210 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.555990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.12.78.1:38099)
0809 20:24:06.561597 (+ 5607us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:06.561607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:06.562138 (+ 531us) server_negotiation.cc:407] Connection header received
0809 20:24:06.562209 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:06.562216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:06.562396 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:06.562527 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:06.564011 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:06.564021 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:06.564037 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:06.564067 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:06.564127 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:06.564153 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:06.564284 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:06.564761 (+ 477us) server_negotiation.cc:299] Negotiation successful
0809 20:24:06.564862 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5435,"thread_start_us":5299,"threads_started":1}
I20260809 20:24:06.565362 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.553691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:35837 (local address 127.0.0.1:49448)
0809 20:24:06.554624 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:06.554667 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:06.554798 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:06.562468 (+ 7670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:06.562476 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:06.562500 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:06.563395 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:06.563400 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:06.563419 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:06.563471 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:06.564012 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:06.564020 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:06.564028 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 20:24:06.564164 (+ 136us) client_negotiation.cc:240] Negotiation successful
0809 20:24:06.564213 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":552,"threads_started":1}
I20260809 20:24:06.566275 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.555702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:38099 (local address 127.0.0.1:38004)
0809 20:24:06.561891 (+ 6189us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:06.561910 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:06.562019 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:06.562943 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:06.562949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:06.562972 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:06.563701 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:06.563705 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:06.563719 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:06.563767 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:06.564347 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:06.564350 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:06.564353 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 20:24:06.564517 (+ 164us) client_negotiation.cc:240] Negotiation successful
0809 20:24:06.564594 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6068,"thread_start_us":5917,"threads_started":1}
I20260809 20:24:06.568454 17812 tablet_service.cc:1467] Processing CreateTablet for tablet 564c403f410b4d1b83c361edc9195d4a (DEFAULT_TABLE table=test-table [id=75ecb98fa29241c0a77c0f6ab1b076f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:06.569795 17812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564c403f410b4d1b83c361edc9195d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:06.569463 17946 tablet_service.cc:1467] Processing CreateTablet for tablet 564c403f410b4d1b83c361edc9195d4a (DEFAULT_TABLE table=test-table [id=75ecb98fa29241c0a77c0f6ab1b076f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:06.570047 17673 tablet_service.cc:1467] Processing CreateTablet for tablet 564c403f410b4d1b83c361edc9195d4a (DEFAULT_TABLE table=test-table [id=75ecb98fa29241c0a77c0f6ab1b076f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:06.570880 17946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564c403f410b4d1b83c361edc9195d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:06.571346 17673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564c403f410b4d1b83c361edc9195d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:06.580894 18035 tablet_bootstrap.cc:492] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf: Bootstrap starting.
I20260809 20:24:06.584590 18035 tablet_bootstrap.cc:654] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:06.585589 18036 tablet_bootstrap.cc:492] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b: Bootstrap starting.
I20260809 20:24:06.586148 18035 log.cc:824] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:06.587517 18037 tablet_bootstrap.cc:492] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b: Bootstrap starting.
I20260809 20:24:06.589784 18036 tablet_bootstrap.cc:654] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:06.591084 18036 log.cc:824] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:06.591321 18037 tablet_bootstrap.cc:654] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:06.591429 18035 tablet_bootstrap.cc:492] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf: No bootstrap required, opened a new log
I20260809 20:24:06.591966 18035 ts_tablet_manager.cc:1397] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260809 20:24:06.592692 18037 log.cc:824] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:06.594574 18036 tablet_bootstrap.cc:492] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b: No bootstrap required, opened a new log
I20260809 20:24:06.594856 18036 ts_tablet_manager.cc:1397] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 20:24:06.598908 18037 tablet_bootstrap.cc:492] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b: No bootstrap required, opened a new log
I20260809 20:24:06.599195 18037 ts_tablet_manager.cc:1397] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260809 20:24:06.603019 18035 raft_consensus.cc:348] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da79877ccc4fb088081dece170111b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 35837 } } peers { permanent_uuid: "59d8652f66344301830c5c76d83193cf" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 45059 } } peers { permanent_uuid: "2e552729fb5c47f18b0870d8287bea7b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 38099 } }
I20260809 20:24:06.603673 18035 raft_consensus.cc:374] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:06.603844 18035 raft_consensus.cc:729] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d8652f66344301830c5c76d83193cf, State: Initialized, Role: FOLLOWER
I20260809 20:24:06.604832 18035 consensus_queue.cc:260] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da79877ccc4fb088081dece170111b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 35837 } } peers { permanent_uuid: "59d8652f66344301830c5c76d83193cf" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 45059 } } peers { permanent_uuid: "2e552729fb5c47f18b0870d8287bea7b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 38099 } }
I20260809 20:24:06.607291 18036 raft_consensus.cc:348] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da79877ccc4fb088081dece170111b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 35837 } } peers { permanent_uuid: "59d8652f66344301830c5c76d83193cf" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 45059 } } peers { permanent_uuid: "2e552729fb5c47f18b0870d8287bea7b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 38099 } }
I20260809 20:24:06.607877 18036 raft_consensus.cc:374] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:06.608047 18036 raft_consensus.cc:729] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e552729fb5c47f18b0870d8287bea7b, State: Initialized, Role: FOLLOWER
I20260809 20:24:06.607991 18037 raft_consensus.cc:348] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da79877ccc4fb088081dece170111b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 35837 } } peers { permanent_uuid: "59d8652f66344301830c5c76d83193cf" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 45059 } } peers { permanent_uuid: "2e552729fb5c47f18b0870d8287bea7b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 38099 } }
I20260809 20:24:06.608438 18037 raft_consensus.cc:374] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:06.608739 18037 raft_consensus.cc:729] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0da79877ccc4fb088081dece170111b, State: Initialized, Role: FOLLOWER
I20260809 20:24:06.609023 18036 consensus_queue.cc:260] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da79877ccc4fb088081dece170111b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 35837 } } peers { permanent_uuid: "59d8652f66344301830c5c76d83193cf" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 45059 } } peers { permanent_uuid: "2e552729fb5c47f18b0870d8287bea7b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 38099 } }
I20260809 20:24:06.609608 18037 consensus_queue.cc:260] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da79877ccc4fb088081dece170111b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 35837 } } peers { permanent_uuid: "59d8652f66344301830c5c76d83193cf" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 45059 } } peers { permanent_uuid: "2e552729fb5c47f18b0870d8287bea7b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 38099 } }
I20260809 20:24:06.612349 17881 heartbeater.cc:503] Master 127.12.78.62:44219 was elected leader, sending a full tablet report...
I20260809 20:24:06.613523 18035 ts_tablet_manager.cc:1428] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf: Time spent starting tablet: real 0.021s user 0.013s sys 0.008s
I20260809 20:24:06.615098 18037 ts_tablet_manager.cc:1428] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260809 20:24:06.615597 18017 heartbeater.cc:503] Master 127.12.78.62:44219 was elected leader, sending a full tablet report...
I20260809 20:24:06.619436 17744 heartbeater.cc:503] Master 127.12.78.62:44219 was elected leader, sending a full tablet report...
I20260809 20:24:06.620002 18036 ts_tablet_manager.cc:1428] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b: Time spent starting tablet: real 0.025s user 0.014s sys 0.009s
I20260809 20:24:06.640110 18041 raft_consensus.cc:482] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:24:06.640470 18041 raft_consensus.cc:504] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da79877ccc4fb088081dece170111b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 35837 } } peers { permanent_uuid: "59d8652f66344301830c5c76d83193cf" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 45059 } } peers { permanent_uuid: "2e552729fb5c47f18b0870d8287bea7b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 38099 } }
I20260809 20:24:06.642318 18041 leader_election.cc:288] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0da79877ccc4fb088081dece170111b (127.12.78.3:35837), 2e552729fb5c47f18b0870d8287bea7b (127.12.78.1:38099)
I20260809 20:24:06.645947 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.642917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:35505 (local address 127.12.78.1:38099)
0809 20:24:06.643156 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:06.643162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:06.643935 (+ 773us) server_negotiation.cc:407] Connection header received
0809 20:24:06.644023 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:06.644028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:06.644088 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:06.644188 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:06.645253 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:06.645259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:06.645264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:06.645283 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:06.645322 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:06.645338 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:06.645464 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:06.645582 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 20:24:06.645670 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 20:24:06.645941 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.642839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:33463 (local address 127.12.78.3:35837)
0809 20:24:06.643047 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:06.643056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:06.643656 (+ 600us) server_negotiation.cc:407] Connection header received
0809 20:24:06.643817 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:06.643821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:06.643873 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:06.644001 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:06.645253 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:06.645259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:06.645264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:06.645283 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:06.645322 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:06.645337 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:06.645472 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:06.645625 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 20:24:06.645682 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 20:24:06.646245 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.642593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:35837 (local address 127.12.78.2:33463)
0809 20:24:06.643499 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:06.643525 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:06.643691 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:06.644047 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:06.644051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:06.644070 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:06.644980 (+ 910us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:06.644985 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:06.645005 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:06.645067 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:06.645446 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:06.645451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:06.645456 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 20:24:06.645639 (+ 183us) client_negotiation.cc:240] Negotiation successful
0809 20:24:06.645704 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":22,"thread_start_us":492,"threads_started":1}
I20260809 20:24:06.646453 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.642593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:38099 (local address 127.12.78.2:35505)
0809 20:24:06.643833 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:06.643846 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:06.643922 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:06.644231 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:06.644238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:06.644261 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:06.645065 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:06.645068 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:06.645080 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:06.645112 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:06.645446 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:06.645451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:06.645456 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 20:24:06.645613 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 20:24:06.645690 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1110,"mutex_wait_us":581,"thread_start_us":300,"threads_started":1}
I20260809 20:24:06.647301 17967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564c403f410b4d1b83c361edc9195d4a" candidate_uuid: "59d8652f66344301830c5c76d83193cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0da79877ccc4fb088081dece170111b" is_pre_election: true
I20260809 20:24:06.647415 17695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564c403f410b4d1b83c361edc9195d4a" candidate_uuid: "59d8652f66344301830c5c76d83193cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e552729fb5c47f18b0870d8287bea7b" is_pre_election: true
I20260809 20:24:06.647863 17967 raft_consensus.cc:2454] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59d8652f66344301830c5c76d83193cf in term 0.
I20260809 20:24:06.647990 17695 raft_consensus.cc:2454] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59d8652f66344301830c5c76d83193cf in term 0.
I20260809 20:24:06.648648 17767 leader_election.cc:302] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59d8652f66344301830c5c76d83193cf, e0da79877ccc4fb088081dece170111b; no voters:
I20260809 20:24:06.649056 18041 raft_consensus.cc:2781] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:24:06.649214 18041 raft_consensus.cc:482] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:24:06.649326 18041 raft_consensus.cc:3037] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:06.651520 18041 raft_consensus.cc:504] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da79877ccc4fb088081dece170111b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 35837 } } peers { permanent_uuid: "59d8652f66344301830c5c76d83193cf" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 45059 } } peers { permanent_uuid: "2e552729fb5c47f18b0870d8287bea7b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 38099 } }
I20260809 20:24:06.652212 18041 leader_election.cc:288] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [CANDIDATE]: Term 1 election: Requested vote from peers e0da79877ccc4fb088081dece170111b (127.12.78.3:35837), 2e552729fb5c47f18b0870d8287bea7b (127.12.78.1:38099)
I20260809 20:24:06.652704 17967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564c403f410b4d1b83c361edc9195d4a" candidate_uuid: "59d8652f66344301830c5c76d83193cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0da79877ccc4fb088081dece170111b"
I20260809 20:24:06.652781 17695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "564c403f410b4d1b83c361edc9195d4a" candidate_uuid: "59d8652f66344301830c5c76d83193cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e552729fb5c47f18b0870d8287bea7b"
I20260809 20:24:06.652913 17967 raft_consensus.cc:3037] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:06.653024 17695 raft_consensus.cc:3037] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:06.655594 17967 raft_consensus.cc:2454] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59d8652f66344301830c5c76d83193cf in term 1.
I20260809 20:24:06.655617 17695 raft_consensus.cc:2454] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59d8652f66344301830c5c76d83193cf in term 1.
I20260809 20:24:06.656059 17766 leader_election.cc:302] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e552729fb5c47f18b0870d8287bea7b, 59d8652f66344301830c5c76d83193cf, e0da79877ccc4fb088081dece170111b; no voters:
I20260809 20:24:06.656440 18041 raft_consensus.cc:2781] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:06.656948 18041 raft_consensus.cc:686] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [term 1 LEADER]: Becoming Leader. State: Replica: 59d8652f66344301830c5c76d83193cf, State: Running, Role: LEADER
I20260809 20:24:06.657491 18041 consensus_queue.cc:237] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da79877ccc4fb088081dece170111b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 35837 } } peers { permanent_uuid: "59d8652f66344301830c5c76d83193cf" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 45059 } } peers { permanent_uuid: "2e552729fb5c47f18b0870d8287bea7b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 38099 } }
I20260809 20:24:06.662683 17541 catalog_manager.cc:5495] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 59d8652f66344301830c5c76d83193cf (127.12.78.2). New cstate: current_term: 1 leader_uuid: "59d8652f66344301830c5c76d83193cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da79877ccc4fb088081dece170111b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 35837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59d8652f66344301830c5c76d83193cf" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 45059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e552729fb5c47f18b0870d8287bea7b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 38099 } health_report { overall_health: UNKNOWN } } }
I20260809 20:24:06.687736 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.685542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.12.78.2:45059)
0809 20:24:06.685743 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:06.685747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:06.685767 (+ 20us) server_negotiation.cc:407] Connection header received
0809 20:24:06.685932 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:06.685937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:06.686028 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:06.686118 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:06.687069 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:06.687074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:06.687078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:06.687093 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:06.687124 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:06.687136 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:06.687238 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:06.687331 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 20:24:06.687376 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 20:24:06.696831 17695 raft_consensus.cc:1261] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b [term 1 FOLLOWER]: Refusing update from remote peer 59d8652f66344301830c5c76d83193cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:06.696831 17967 raft_consensus.cc:1261] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b [term 1 FOLLOWER]: Refusing update from remote peer 59d8652f66344301830c5c76d83193cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:06.698000 18046 consensus_queue.cc:1035] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e552729fb5c47f18b0870d8287bea7b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:06.698401 18041 consensus_queue.cc:1035] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0da79877ccc4fb088081dece170111b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 35837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:06.711145 18049 mvcc.cc:205] Tried to move back new op lower bound from 7316713663258009600 to 7316713663113441280. Current Snapshot: MvccSnapshot[applied={T|T < 7316713663258009600}]
I20260809 20:24:06.725849 17543 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:06.726163 17543 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:06.730005 17543 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c14b763849644488f716b487d5c4b39: Sending DeleteTablet for 3 replicas of tablet 564c403f410b4d1b83c361edc9195d4a
I20260809 20:24:06.731390 17946 tablet_service.cc:1514] Processing DeleteTablet for tablet 564c403f410b4d1b83c361edc9195d4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:06 UTC) from {username='slave'} at 127.0.0.1:49448
I20260809 20:24:06.732147 17673 tablet_service.cc:1514] Processing DeleteTablet for tablet 564c403f410b4d1b83c361edc9195d4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:06 UTC) from {username='slave'} at 127.0.0.1:38004
I20260809 20:24:06.732276 18058 tablet_replica.cc:330] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b: stopping tablet replica
I20260809 20:24:06.732515 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 17607
I20260809 20:24:06.732960 18058 raft_consensus.cc:2229] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:06.733191 17812 tablet_service.cc:1514] Processing DeleteTablet for tablet 564c403f410b4d1b83c361edc9195d4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:06 UTC) from {username='slave'} at 127.0.0.1:36606
I20260809 20:24:06.733668 18058 raft_consensus.cc:2258] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:06.736132 18058 ts_tablet_manager.cc:1905] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:06.745048 18061 tablet_replica.cc:330] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf: stopping tablet replica
I20260809 20:24:06.744874 18058 ts_tablet_manager.cc:1918] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:06.745544 18058 log.cc:1196] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/564c403f410b4d1b83c361edc9195d4a
I20260809 20:24:06.745532 18061 raft_consensus.cc:2229] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:24:06.746207 18058 ts_tablet_manager.cc:1939] T 564c403f410b4d1b83c361edc9195d4a P e0da79877ccc4fb088081dece170111b: Deleting consensus metadata
I20260809 20:24:06.746222 18061 raft_consensus.cc:2258] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:06.746853 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.734145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.12.78.1:38099)
0809 20:24:06.734344 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:06.734351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:06.743685 (+ 9334us) server_negotiation.cc:407] Connection header received
0809 20:24:06.744146 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:06.744153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:06.744276 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:06.744356 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:06.745917 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:06.745923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:06.745928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:06.745949 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:06.745987 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:06.746003 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:06.746107 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:06.746471 (+ 364us) server_negotiation.cc:299] Negotiation successful
0809 20:24:06.746541 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 20:24:06.748091 17739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:06.748654 18061 ts_tablet_manager.cc:1905] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:06.754514 17529 catalog_manager.cc:4841] TS e0da79877ccc4fb088081dece170111b (127.12.78.3:35837): tablet 564c403f410b4d1b83c361edc9195d4a (table test-table [id=75ecb98fa29241c0a77c0f6ab1b076f4]) successfully deleted
I20260809 20:24:06.758411 18061 ts_tablet_manager.cc:1918] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:06.758594 18061 log.cc:1196] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/564c403f410b4d1b83c361edc9195d4a
I20260809 20:24:06.759092 18061 ts_tablet_manager.cc:1939] T 564c403f410b4d1b83c361edc9195d4a P 59d8652f66344301830c5c76d83193cf: Deleting consensus metadata
I20260809 20:24:06.760690 17526 catalog_manager.cc:4841] TS 59d8652f66344301830c5c76d83193cf (127.12.78.2:45059): tablet 564c403f410b4d1b83c361edc9195d4a (table test-table [id=75ecb98fa29241c0a77c0f6ab1b076f4]) successfully deleted
I20260809 20:24:06.858358 18059 tablet_replica.cc:330] T 564c403f410b4d1b83c361edc9195d4a P 2e552729fb5c47f18b0870d8287bea7b: stopping tablet replica
I20260809 20:24:06.858779 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 17607
W20260809 20:24:06.878214 17528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.78.1:38099 (error 108)
I20260809 20:24:06.878449 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 17748
I20260809 20:24:06.880374 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.879861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:38099 (local address 127.0.0.1:38018)
0809 20:24:06.880048 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:06.880190 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:38099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 20:24:06.880633 17528 catalog_manager.cc:4584] TS 2e552729fb5c47f18b0870d8287bea7b (127.12.78.1:38099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 564c403f410b4d1b83c361edc9195d4a: Network error: Client connection negotiation failed: client connection to 127.12.78.1:38099: connect: Connection refused (error 111)
I20260809 20:24:06.881896 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.879569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.12.78.2:45059)
0809 20:24:06.879758 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:06.879764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:06.879784 (+ 20us) server_negotiation.cc:407] Connection header received
0809 20:24:06.879877 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:06.879883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:06.879944 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:06.880048 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:06.881257 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:06.881267 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:06.881272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:06.881293 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:06.881331 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:06.881347 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:06.881485 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:06.881534 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 20:24:06.881588 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 20:24:06.882647 17876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:06.934594 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.934014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:38099 (local address 127.0.0.1:38028)
0809 20:24:06.934317 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:06.934431 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:38099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260809 20:24:06.935590 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.935282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:38099 (local address 127.0.0.1:38034)
0809 20:24:06.935411 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:06.935478 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:38099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 20:24:06.993220 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 17748
I20260809 20:24:06.996372 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.995795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:38099 (local address 127.0.0.1:38050)
0809 20:24:06.996030 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:06.996164 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:38099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 20:24:06.997574 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:06.997129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:38099 (local address 127.0.0.1:38052)
0809 20:24:06.997305 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:06.997412 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:38099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 20:24:07.016026 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 17884
I20260809 20:24:07.024048 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:07.016825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.12.78.3:35837)
0809 20:24:07.017003 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:07.017010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:07.017080 (+ 70us) server_negotiation.cc:407] Connection header received
0809 20:24:07.017314 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:07.017319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:07.017396 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:07.017501 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:07.023329 (+ 5828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:07.023334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:07.023337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:07.023354 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:07.023389 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:07.023402 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:07.023557 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:07.023667 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 20:24:07.023728 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 20:24:07.024739 18011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:07.078475 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:07.077934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:38099 (local address 127.0.0.1:38060)
0809 20:24:07.078233 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:07.078331 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:38099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260809 20:24:07.079548 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:07.079145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:38099 (local address 127.0.0.1:38062)
0809 20:24:07.079362 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:07.079431 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:38099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 20:24:07.112524 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 17884
I20260809 20:24:07.130926 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 17509
I20260809 20:24:07.132097 17576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:07.240063 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 17509
2026-08-09T20:24:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2693 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 20:24:07.284188 12600 mini_oidc.cc:152] Starting JWKS server
I20260809 20:24:07.287132 12600 webserver.cc:460] Webserver started at https://127.0.0.1:44167/ using document root <none> and password file <none>
I20260809 20:24:07.287487 12600 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 20:24:07.288163 12600 webserver.cc:460] Webserver started at http://127.0.0.1:42319/ using document root <none> and password file <none>
2026-08-09T20:24:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:24:07Z Disabled control of system clock
I20260809 20:24:07.319528 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:35405
--webserver_interface=127.12.78.62
--webserver_port=0
--builtin_ntp_servers=127.12.78.20:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:35405
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44167/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/testchainca.cert with env {}
W20260809 20:24:07.649425 18072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:07.649718 18072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:07.649807 18072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:07.656996 18072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:07.657105 18072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:07.657143 18072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:07.657202 18072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:07.657246 18072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 20:24:07.657302 18072 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 20:24:07.657343 18072 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44167/jwks.json
I20260809 20:24:07.667244 18072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:35405
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:35405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44167/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:07.668526 18072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:07.669800 18072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:07.682169 18077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:07.682192 18078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:07.682435 18080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:07.683172 18072 server_base.cc:1034] running on GCE node
I20260809 20:24:07.684005 18072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:07.685518 18072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:07.686782 18072 hybrid_clock.cc:648] HybridClock initialized: now 1786307047686729 us; error 63 us; skew 500 ppm
I20260809 20:24:07.687171 18072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:07.689491 18072 webserver.cc:460] Webserver started at http://127.12.78.62:36449/ using document root <none> and password file <none>
I20260809 20:24:07.690183 18072 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:07.690315 18072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:07.690630 18072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:07.693783 18072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "1705517033e0407e8346a591fa4b70bc"
format_stamp: "Formatted at 2026-08-09 20:24:07 on dist-test-slave-rnrw"
server_key: "fe83c7f61dcf9a8d4cf9e70f529e94c8"
server_key_iv: "d9f7c2a872ef7728e5b1955ec3934109"
server_key_version: "encryptionkey@0"
I20260809 20:24:07.694591 18072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "1705517033e0407e8346a591fa4b70bc"
format_stamp: "Formatted at 2026-08-09 20:24:07 on dist-test-slave-rnrw"
server_key: "fe83c7f61dcf9a8d4cf9e70f529e94c8"
server_key_iv: "d9f7c2a872ef7728e5b1955ec3934109"
server_key_version: "encryptionkey@0"
I20260809 20:24:07.700241 18072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260809 20:24:07.703558 18086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:07.704898 18072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 20:24:07.705082 18072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "1705517033e0407e8346a591fa4b70bc"
format_stamp: "Formatted at 2026-08-09 20:24:07 on dist-test-slave-rnrw"
server_key: "fe83c7f61dcf9a8d4cf9e70f529e94c8"
server_key_iv: "d9f7c2a872ef7728e5b1955ec3934109"
server_key_version: "encryptionkey@0"
I20260809 20:24:07.705289 18072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:07.728308 18072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:07.729206 18072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:07.729542 18072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:07.747823 18072 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:35405
I20260809 20:24:07.747905 18142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:35405 every 8 connection(s)
I20260809 20:24:07.749243 18072 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:24:07.752946 18143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:07.758307 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 18072
I20260809 20:24:07.758523 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:24:07.759195 12600 external_mini_cluster.cc:1428] Setting key d4a9eddc37e5b0a766d3cd2578b4bee2
I20260809 20:24:07.763958 18143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc: Bootstrap starting.
I20260809 20:24:07.767045 18143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:07.768044 18143 log.cc:824] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:07.770771 18143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc: No bootstrap required, opened a new log
I20260809 20:24:07.772522 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:07.761199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.12.78.62:35405)
0809 20:24:07.762202 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:07.762213 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:07.762254 (+ 41us) server_negotiation.cc:407] Connection header received
0809 20:24:07.763931 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:07.763956 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:07.764436 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:07.764858 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:07.765422 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:07.766677 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:07.768065 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:07.768861 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:07.769069 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:07.770282 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:07.770290 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:07.770302 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:07.770835 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:07.770943 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:07.770977 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:07.771096 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:07.771319 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 20:24:07.771621 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":328,"threads_started":1}
I20260809 20:24:07.776356 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:24:07.779153 18143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1705517033e0407e8346a591fa4b70bc" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35405 } }
I20260809 20:24:07.779624 18143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:07.779733 18143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1705517033e0407e8346a591fa4b70bc, State: Initialized, Role: FOLLOWER
I20260809 20:24:07.780270 18143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1705517033e0407e8346a591fa4b70bc" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35405 } }
I20260809 20:24:07.780507 18143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:24:07.780676 18143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:24:07.780851 18143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:07.782996 18143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1705517033e0407e8346a591fa4b70bc" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35405 } }
I20260809 20:24:07.783473 18143 leader_election.cc:302] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1705517033e0407e8346a591fa4b70bc; no voters:
I20260809 20:24:07.784144 18143 leader_election.cc:288] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:24:07.784282 18148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:07.785144 18148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [term 1 LEADER]: Becoming Leader. State: Replica: 1705517033e0407e8346a591fa4b70bc, State: Running, Role: LEADER
I20260809 20:24:07.785725 18148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1705517033e0407e8346a591fa4b70bc" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35405 } }
I20260809 20:24:07.786510 18143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:07.789244 18150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1705517033e0407e8346a591fa4b70bc. Latest consensus state: current_term: 1 leader_uuid: "1705517033e0407e8346a591fa4b70bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1705517033e0407e8346a591fa4b70bc" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35405 } } }
I20260809 20:24:07.789742 18150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:07.789556 18149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1705517033e0407e8346a591fa4b70bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1705517033e0407e8346a591fa4b70bc" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35405 } } }
I20260809 20:24:07.789952 18149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:07.791718 18156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:07.796973 18156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:07.805913 18156 catalog_manager.cc:1347] Generated new cluster ID: f89e52e61c644ee79cd68ac85c6d2139
I20260809 20:24:07.806066 18156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:07.826891 18156 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:24:07.828209 18156 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:07.841005 18156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1705517033e0407e8346a591fa4b70bc: Generated new TSK 0
I20260809 20:24:07.841886 18156 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 20:24:07.880759 18173 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 20:24:07.880981 18173 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 20:24:07.882236 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:07.858088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.12.78.62:35405)
0809 20:24:07.858280 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:07.858287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:07.858720 (+ 433us) server_negotiation.cc:407] Connection header received
0809 20:24:07.859045 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:07.859051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:07.859143 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:07.859310 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0809 20:24:07.859808 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:07.860985 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:07.862422 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:07.863132 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:07.863257 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:07.863775 (+ 518us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 20:24:07.881535 (+ 17760us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 20:24:07.881944 (+ 409us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260809 20:24:07.882368 18146 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 20:24:07.883759 18172 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:07.857946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:35405 (local address 127.0.0.1:54886)
0809 20:24:07.858611 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:07.858628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:07.858776 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:07.859324 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:07.859327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:07.859337 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0809 20:24:07.859674 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:07.859682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:07.861134 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:07.861138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:07.862284 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:07.862295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:07.863297 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:07.863302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:07.863426 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:07.863484 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 20:24:07.881830 (+ 18346us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 20:24:07.882124 (+ 294us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.78.62:35405: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":340,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.78.62:35405: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44167/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 20:24:07.885288 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 18072
I20260809 20:24:07.886844 18138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:07.981523 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 18072
2026-08-09T20:24:08Z chronyd exiting
I20260809 20:24:08.034154 12600 test_util.cc:175] -----------------------------------------------
I20260809 20:24:08.034269 12600 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786307001768029-12600-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (754 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 20:24:08.038053 12600 mini_oidc.cc:152] Starting JWKS server
I20260809 20:24:08.039935 12600 webserver.cc:460] Webserver started at https://127.0.0.1:38037/ using document root <none> and password file <none>
I20260809 20:24:08.040177 12600 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 20:24:08.040877 12600 webserver.cc:460] Webserver started at http://127.0.0.1:41161/ using document root <none> and password file <none>
2026-08-09T20:24:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:24:08Z Disabled control of system clock
I20260809 20:24:08.070201 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:35615
--webserver_interface=127.12.78.62
--webserver_port=0
--builtin_ntp_servers=127.12.78.20:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:35615
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38037/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/testchainca.cert with env {}
W20260809 20:24:08.395697 18180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:08.396039 18180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:08.396198 18180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:08.403317 18180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:08.403468 18180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:08.403561 18180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:08.403664 18180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:08.403746 18180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 20:24:08.403836 18180 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 20:24:08.403903 18180 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38037/jwks.json
I20260809 20:24:08.413779 18180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:35615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:35615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38037/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:08.415122 18180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:08.416481 18180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:08.428020 18186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:08.428071 18189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:08.428094 18187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:08.429514 18180 server_base.cc:1034] running on GCE node
I20260809 20:24:08.430359 18180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:08.431787 18180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:08.433008 18180 hybrid_clock.cc:648] HybridClock initialized: now 1786307048432970 us; error 48 us; skew 500 ppm
I20260809 20:24:08.433367 18180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:08.435892 18180 webserver.cc:460] Webserver started at http://127.12.78.62:37573/ using document root <none> and password file <none>
I20260809 20:24:08.436669 18180 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:08.436805 18180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:08.437139 18180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:08.440596 18180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "4336b8fe9621465484657108a3fafd95"
format_stamp: "Formatted at 2026-08-09 20:24:08 on dist-test-slave-rnrw"
server_key: "e9824b90b01b9a95fe399e4f30d9e063"
server_key_iv: "c5abac169c3ffc118ffbf6ba0efb691b"
server_key_version: "encryptionkey@0"
I20260809 20:24:08.441429 18180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "4336b8fe9621465484657108a3fafd95"
format_stamp: "Formatted at 2026-08-09 20:24:08 on dist-test-slave-rnrw"
server_key: "e9824b90b01b9a95fe399e4f30d9e063"
server_key_iv: "c5abac169c3ffc118ffbf6ba0efb691b"
server_key_version: "encryptionkey@0"
I20260809 20:24:08.446578 18180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:24:08.450098 18195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:08.451398 18180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 20:24:08.451573 18180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "4336b8fe9621465484657108a3fafd95"
format_stamp: "Formatted at 2026-08-09 20:24:08 on dist-test-slave-rnrw"
server_key: "e9824b90b01b9a95fe399e4f30d9e063"
server_key_iv: "c5abac169c3ffc118ffbf6ba0efb691b"
server_key_version: "encryptionkey@0"
I20260809 20:24:08.451790 18180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:08.470111 18180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:08.470937 18180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:08.471261 18180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:08.489814 18180 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:35615
I20260809 20:24:08.489827 18248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:35615 every 8 connection(s)
I20260809 20:24:08.491210 18180 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:24:08.495110 18249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:08.498680 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 18180
I20260809 20:24:08.498912 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:24:08.499555 12600 external_mini_cluster.cc:1428] Setting key c3a861ba9a31b0bfd413b4651af3ca49
I20260809 20:24:08.508000 18249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95: Bootstrap starting.
I20260809 20:24:08.512764 18249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:08.514147 18249 log.cc:824] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:08.517659 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:08.501665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.12.78.62:35615)
0809 20:24:08.502835 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:08.502851 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:08.502909 (+ 58us) server_negotiation.cc:407] Connection header received
0809 20:24:08.506083 (+ 3174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:08.506116 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:08.506764 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:08.507226 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:08.507798 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:08.510858 (+ 3060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:08.512290 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:08.513465 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:08.513655 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:08.514887 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:08.514903 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:08.514930 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:08.515509 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:08.515642 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:08.515678 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:08.515785 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:08.516012 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 20:24:08.516342 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":360,"threads_started":1}
I20260809 20:24:08.517712 18249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95: No bootstrap required, opened a new log
I20260809 20:24:08.521664 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:24:08.526075 18249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4336b8fe9621465484657108a3fafd95" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35615 } }
I20260809 20:24:08.526445 18249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:08.526558 18249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4336b8fe9621465484657108a3fafd95, State: Initialized, Role: FOLLOWER
I20260809 20:24:08.527092 18249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4336b8fe9621465484657108a3fafd95" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35615 } }
I20260809 20:24:08.527328 18249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:24:08.527489 18249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:24:08.527650 18249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:08.529865 18249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4336b8fe9621465484657108a3fafd95" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35615 } }
I20260809 20:24:08.530346 18249 leader_election.cc:302] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4336b8fe9621465484657108a3fafd95; no voters:
I20260809 20:24:08.530992 18249 leader_election.cc:288] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:24:08.531143 18254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:08.531948 18254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [term 1 LEADER]: Becoming Leader. State: Replica: 4336b8fe9621465484657108a3fafd95, State: Running, Role: LEADER
I20260809 20:24:08.532490 18254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4336b8fe9621465484657108a3fafd95" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35615 } }
I20260809 20:24:08.533198 18249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:08.535620 18256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4336b8fe9621465484657108a3fafd95. Latest consensus state: current_term: 1 leader_uuid: "4336b8fe9621465484657108a3fafd95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4336b8fe9621465484657108a3fafd95" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35615 } } }
I20260809 20:24:08.535670 18255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4336b8fe9621465484657108a3fafd95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4336b8fe9621465484657108a3fafd95" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35615 } } }
I20260809 20:24:08.536233 18255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:08.536732 18256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:08.541702 18264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:08.546749 18264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:08.554800 18264 catalog_manager.cc:1347] Generated new cluster ID: aca23c864cef4179b81f8ae189cda753
I20260809 20:24:08.554911 18264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:08.584784 18264 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:24:08.586624 18264 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:08.598199 18264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4336b8fe9621465484657108a3fafd95: Generated new TSK 0
I20260809 20:24:08.599246 18264 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 20:24:08.631805 18280 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 20:24:08.631942 18280 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 20:24:08.633054 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:08.612991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.12.78.62:35615)
0809 20:24:08.613228 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:08.613237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:08.613674 (+ 437us) server_negotiation.cc:407] Connection header received
0809 20:24:08.613916 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:08.613919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:08.613998 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:08.614126 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0809 20:24:08.614693 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:08.615401 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:08.617383 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:08.617804 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:08.617906 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:08.618479 (+ 573us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 20:24:08.632454 (+ 13975us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 20:24:08.632840 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58800: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260809 20:24:08.633298 18252 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58800: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260809 20:24:08.633402 18279 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:08.612910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:35615 (local address 127.0.0.1:58800)
0809 20:24:08.613565 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:08.613585 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:08.613742 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:08.614165 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:08.614172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:08.614189 (+ 17us) client_negotiation.cc:189] Negotiated authn=JWT
0809 20:24:08.614520 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:08.614532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:08.615540 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:08.615547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:08.617244 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:08.617259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:08.617974 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:08.617981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:08.618182 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:08.618228 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 20:24:08.632716 (+ 14488us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 20:24:08.632991 (+ 275us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.78.62:35615: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":330,"threads_started":1}
I20260809 20:24:08.634732 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 18180
I20260809 20:24:08.635918 18244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:08.732453 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 18180
2026-08-09T20:24:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (751 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 20:24:08.789649 12600 mini_oidc.cc:152] Starting JWKS server
I20260809 20:24:08.791684 12600 webserver.cc:460] Webserver started at https://127.0.0.1:39941/ using document root <none> and password file <none>
I20260809 20:24:08.791931 12600 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 20:24:08.792620 12600 webserver.cc:460] Webserver started at http://127.0.0.1:46211/ using document root <none> and password file <none>
2026-08-09T20:24:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:24:08Z Disabled control of system clock
I20260809 20:24:08.824276 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:33907
--webserver_interface=127.12.78.62
--webserver_port=0
--builtin_ntp_servers=127.12.78.20:33699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:33907
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39941/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 20:24:09.159118 18288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:09.159471 18288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:09.159636 18288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:09.166972 18288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:09.167119 18288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:09.167207 18288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:09.167308 18288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:09.167394 18288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 20:24:09.167498 18288 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 20:24:09.167573 18288 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39941/jwks.json
I20260809 20:24:09.176892 18288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:33699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:33907
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:33907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39941/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:09.178251 18288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:09.180017 18288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:09.191545 18293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:09.191532 18296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:09.191485 18294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:09.192514 18288 server_base.cc:1034] running on GCE node
I20260809 20:24:09.193457 18288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:09.194885 18288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:09.196117 18288 hybrid_clock.cc:648] HybridClock initialized: now 1786307049196074 us; error 36 us; skew 500 ppm
I20260809 20:24:09.196447 18288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:09.198593 18288 webserver.cc:460] Webserver started at http://127.12.78.62:37741/ using document root <none> and password file <none>
I20260809 20:24:09.199270 18288 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:09.199397 18288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:09.199755 18288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:09.202622 18288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "7d6364cff14b4a13a810460d5807d75b"
format_stamp: "Formatted at 2026-08-09 20:24:09 on dist-test-slave-rnrw"
server_key: "8fba44ac5e10f25582c9d8653c5e79d0"
server_key_iv: "118c8a9e05a8181499570d8a7ccef3c8"
server_key_version: "encryptionkey@0"
I20260809 20:24:09.203424 18288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "7d6364cff14b4a13a810460d5807d75b"
format_stamp: "Formatted at 2026-08-09 20:24:09 on dist-test-slave-rnrw"
server_key: "8fba44ac5e10f25582c9d8653c5e79d0"
server_key_iv: "118c8a9e05a8181499570d8a7ccef3c8"
server_key_version: "encryptionkey@0"
I20260809 20:24:09.208415 18288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 20:24:09.211736 18303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:09.213063 18288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:09.213246 18288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "7d6364cff14b4a13a810460d5807d75b"
format_stamp: "Formatted at 2026-08-09 20:24:09 on dist-test-slave-rnrw"
server_key: "8fba44ac5e10f25582c9d8653c5e79d0"
server_key_iv: "118c8a9e05a8181499570d8a7ccef3c8"
server_key_version: "encryptionkey@0"
I20260809 20:24:09.213459 18288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:09.227015 18288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:09.227846 18288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:09.228164 18288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:09.248384 18288 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:33907
I20260809 20:24:09.248406 18358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:33907 every 8 connection(s)
I20260809 20:24:09.249859 18288 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:24:09.253724 18359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:09.254252 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 18288
I20260809 20:24:09.254444 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:24:09.255106 12600 external_mini_cluster.cc:1428] Setting key a5906e86743ad87fa8e3f24f167453fa
I20260809 20:24:09.268698 18359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b: Bootstrap starting.
I20260809 20:24:09.270838 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:09.257154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.12.78.62:33907)
0809 20:24:09.258302 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:09.258327 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:09.258379 (+ 52us) server_negotiation.cc:407] Connection header received
0809 20:24:09.260491 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:09.260520 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:09.261172 (+ 652us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:09.261625 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:09.262155 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:09.264017 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:09.265434 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:09.266583 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:09.266761 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:09.267954 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:09.267964 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:09.267983 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:09.268748 (+ 765us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:09.268853 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:09.268878 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:09.268991 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:09.269188 (+ 197us) server_negotiation.cc:299] Negotiation successful
0809 20:24:09.269504 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":377,"threads_started":1}
I20260809 20:24:09.272624 18359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:09.273908 18359 log.cc:824] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:09.274950 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:24:09.277277 18359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b: No bootstrap required, opened a new log
I20260809 20:24:09.285223 18359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d6364cff14b4a13a810460d5807d75b" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33907 } }
I20260809 20:24:09.285573 18359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:09.285652 18359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d6364cff14b4a13a810460d5807d75b, State: Initialized, Role: FOLLOWER
I20260809 20:24:09.286214 18359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d6364cff14b4a13a810460d5807d75b" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33907 } }
I20260809 20:24:09.286415 18359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:24:09.286515 18359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:24:09.286643 18359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:09.288861 18359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d6364cff14b4a13a810460d5807d75b" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33907 } }
I20260809 20:24:09.289410 18359 leader_election.cc:302] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d6364cff14b4a13a810460d5807d75b; no voters:
I20260809 20:24:09.290092 18359 leader_election.cc:288] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:24:09.290266 18364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:09.290974 18364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [term 1 LEADER]: Becoming Leader. State: Replica: 7d6364cff14b4a13a810460d5807d75b, State: Running, Role: LEADER
I20260809 20:24:09.291509 18364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d6364cff14b4a13a810460d5807d75b" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33907 } }
I20260809 20:24:09.292212 18359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:09.294226 18365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d6364cff14b4a13a810460d5807d75b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d6364cff14b4a13a810460d5807d75b" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33907 } } }
I20260809 20:24:09.294600 18365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:09.295248 18366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d6364cff14b4a13a810460d5807d75b. Latest consensus state: current_term: 1 leader_uuid: "7d6364cff14b4a13a810460d5807d75b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d6364cff14b4a13a810460d5807d75b" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33907 } } }
I20260809 20:24:09.295651 18366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:09.297308 18373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:09.302727 18373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:09.311467 18373 catalog_manager.cc:1347] Generated new cluster ID: 828d6506199045809bfbbbf0de2210a6
I20260809 20:24:09.311585 18373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:09.325770 18373 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:24:09.327029 18373 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:09.343025 18373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d6364cff14b4a13a810460d5807d75b: Generated new TSK 0
I20260809 20:24:09.343916 18373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:24:09.372924 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:09.364676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.12.78.62:33907)
0809 20:24:09.364889 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:09.364895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:09.365224 (+ 329us) server_negotiation.cc:407] Connection header received
0809 20:24:09.365501 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:09.365505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:09.365576 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:09.365700 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0809 20:24:09.366235 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:09.367096 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:09.369183 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:09.369649 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:09.369755 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:09.370378 (+ 623us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 20:24:09.372256 (+ 1878us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 20:24:09.372630 (+ 374us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36682: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":82}
W20260809 20:24:09.373203 18391 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:09.364459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:33907 (local address 127.0.0.1:36682)
0809 20:24:09.365116 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:09.365137 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:09.365292 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:09.365739 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:09.365745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:09.365757 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0809 20:24:09.366081 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:09.366093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:09.367233 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:09.367239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:09.368987 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:09.369002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:09.369839 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:09.369845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:09.370044 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:09.370087 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 20:24:09.372455 (+ 2368us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 20:24:09.372766 (+ 311us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.78.62:33907: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":344,"threads_started":1}
W20260809 20:24:09.373337 18362 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36682: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260809 20:24:09.374398 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 18288
I20260809 20:24:09.375521 18353 generic_service.cc:196] Checking for leaks (request via RPC)
E20260809 20:24:09.385473 18392 webserver.cc:533] Webserver: sslize failed:
E20260809 20:24:09.385566 18392 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 20:24:09.467350 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 18288
2026-08-09T20:24:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (732 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T20:24:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:24:09Z Disabled control of system clock
I20260809 20:24:09.551508 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:36077
--webserver_interface=127.12.78.62
--webserver_port=0
--builtin_ntp_servers=127.12.78.20:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:36077
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/testchainca.cert with env {}
W20260809 20:24:09.884482 18398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:09.884814 18398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:09.884980 18398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:09.892253 18398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:09.892365 18398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:09.892457 18398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:09.892568 18398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:09.892659 18398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 20:24:09.892743 18398 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 20:24:09.892812 18398 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 20:24:09.902947 18398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:36077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:36077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:09.904286 18398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:09.905608 18398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:09.917804 18404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:09.917883 18403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:09.917887 18406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:09.918920 18398 server_base.cc:1034] running on GCE node
I20260809 20:24:09.919775 18398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:09.921268 18398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:09.922521 18398 hybrid_clock.cc:648] HybridClock initialized: now 1786307049922478 us; error 30 us; skew 500 ppm
I20260809 20:24:09.922866 18398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:09.925357 18398 webserver.cc:460] Webserver started at http://127.12.78.62:35993/ using document root <none> and password file <none>
I20260809 20:24:09.926072 18398 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:09.926192 18398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:09.926508 18398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:09.929476 18398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "85ec372f4f81492fa7e8c5730044cb60"
format_stamp: "Formatted at 2026-08-09 20:24:09 on dist-test-slave-rnrw"
server_key: "4755ffff2dd4e160aeeea540c596ce5a"
server_key_iv: "b1ffc2ca0bc1ce49f9850cdef5cd76d9"
server_key_version: "encryptionkey@0"
I20260809 20:24:09.930302 18398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "85ec372f4f81492fa7e8c5730044cb60"
format_stamp: "Formatted at 2026-08-09 20:24:09 on dist-test-slave-rnrw"
server_key: "4755ffff2dd4e160aeeea540c596ce5a"
server_key_iv: "b1ffc2ca0bc1ce49f9850cdef5cd76d9"
server_key_version: "encryptionkey@0"
I20260809 20:24:09.935240 18398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:24:09.938541 18412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:09.939888 18398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 20:24:09.940064 18398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "85ec372f4f81492fa7e8c5730044cb60"
format_stamp: "Formatted at 2026-08-09 20:24:09 on dist-test-slave-rnrw"
server_key: "4755ffff2dd4e160aeeea540c596ce5a"
server_key_iv: "b1ffc2ca0bc1ce49f9850cdef5cd76d9"
server_key_version: "encryptionkey@0"
I20260809 20:24:09.940253 18398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:09.954352 18398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:09.955147 18398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:09.955422 18398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:09.974077 18398 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:36077
I20260809 20:24:09.974090 18463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:36077 every 8 connection(s)
I20260809 20:24:09.975492 18398 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:24:09.979343 18464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:09.981388 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 18398
I20260809 20:24:09.981598 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:24:09.982229 12600 external_mini_cluster.cc:1428] Setting key 6d7fd5d507fecb4a84c48f6aefbce470
I20260809 20:24:09.991433 18464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60: Bootstrap starting.
I20260809 20:24:09.995244 18464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:09.996306 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:09.984248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.12.78.62:36077)
0809 20:24:09.985397 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:09.985416 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:09.985458 (+ 42us) server_negotiation.cc:407] Connection header received
0809 20:24:09.987096 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:09.987126 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:09.987593 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:09.987935 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:09.988639 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:09.989938 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:09.991737 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:09.992433 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:09.992634 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:09.993923 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:09.993930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:09.993943 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:09.994477 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:09.994572 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:09.994594 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:09.994692 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:09.994919 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 20:24:09.995284 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":478,"threads_started":1}
I20260809 20:24:09.996358 18464 log.cc:824] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:09.999995 18464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60: No bootstrap required, opened a new log
I20260809 20:24:10.000713 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:24:10.009219 18464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ec372f4f81492fa7e8c5730044cb60" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 36077 } }
I20260809 20:24:10.009620 18464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:10.009732 18464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ec372f4f81492fa7e8c5730044cb60, State: Initialized, Role: FOLLOWER
I20260809 20:24:10.010389 18464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ec372f4f81492fa7e8c5730044cb60" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 36077 } }
I20260809 20:24:10.010695 18464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:24:10.010857 18464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:24:10.011041 18464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:10.013324 18464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ec372f4f81492fa7e8c5730044cb60" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 36077 } }
I20260809 20:24:10.013816 18464 leader_election.cc:302] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85ec372f4f81492fa7e8c5730044cb60; no voters:
I20260809 20:24:10.014396 18464 leader_election.cc:288] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:24:10.014537 18469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:10.015311 18469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [term 1 LEADER]: Becoming Leader. State: Replica: 85ec372f4f81492fa7e8c5730044cb60, State: Running, Role: LEADER
I20260809 20:24:10.016176 18469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ec372f4f81492fa7e8c5730044cb60" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 36077 } }
I20260809 20:24:10.016837 18464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:10.019464 18471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85ec372f4f81492fa7e8c5730044cb60. Latest consensus state: current_term: 1 leader_uuid: "85ec372f4f81492fa7e8c5730044cb60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ec372f4f81492fa7e8c5730044cb60" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 36077 } } }
I20260809 20:24:10.020001 18471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:10.019914 18470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85ec372f4f81492fa7e8c5730044cb60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ec372f4f81492fa7e8c5730044cb60" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 36077 } } }
I20260809 20:24:10.020308 18470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:10.021907 18478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:10.027253 18478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:10.035749 18478 catalog_manager.cc:1347] Generated new cluster ID: 24fe83867eef47538769c98b5ba89903
I20260809 20:24:10.035864 18478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:10.061820 18478 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:24:10.063102 18478 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:10.075012 18478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85ec372f4f81492fa7e8c5730044cb60: Generated new TSK 0
I20260809 20:24:10.075909 18478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:24:10.578400 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:10.101996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.12.78.62:36077)
0809 20:24:10.102243 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:10.102250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:10.102619 (+ 369us) server_negotiation.cc:407] Connection header received
0809 20:24:10.102852 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:10.102856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:10.102920 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:10.103040 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0809 20:24:10.103579 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:10.104314 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:10.106346 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:10.106780 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:10.106871 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:10.107412 (+ 541us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 20:24:10.577827 (+470415us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 20:24:10.578160 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35868: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":78}
W20260809 20:24:10.578575 18467 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35868: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 20:24:10.578739 18497 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:10.101889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:36077 (local address 127.0.0.1:35868)
0809 20:24:10.102505 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:10.102527 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:10.102680 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:10.103089 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:10.103096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:10.103110 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0809 20:24:10.103428 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:10.103439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:10.104456 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:10.104462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:10.106213 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:10.106228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:10.106951 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:10.106959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:10.107150 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:10.107192 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 20:24:10.578150 (+470958us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 20:24:10.578405 (+ 255us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.78.62:36077: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":308,"threads_started":1}
I20260809 20:24:10.580080 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 18398
I20260809 20:24:10.581182 18459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:10.684898 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 18398
2026-08-09T20:24:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1202 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (421 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (444 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:24:11 dist-test-slave-rnrw krb5kdc[18512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:24:11 dist-test-slave-rnrw krb5kdc[18512](info): set up 2 sockets
Aug 09 20:24:11 dist-test-slave-rnrw krb5kdc[18512](info): commencing operation
krb5kdc: starting...
W20260809 20:24:12.681147 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:24:12 dist-test-slave-rnrw krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:24:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:24:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:24:12.923753 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40509
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:40509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:13.252696 18526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:13.253057 18526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:13.253224 18526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:13.261849 18526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:13.262008 18526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:13.262113 18526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:13.262221 18526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:13.262307 18526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:24:13.272537 18526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:40509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:13.274011 18526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:13.275350 18526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:13.287169 18531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:13.287078 18535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:13.287114 18532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:13.288196 18526 server_base.cc:1034] running on GCE node
I20260809 20:24:13.289075 18526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:13.290481 18526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:13.291729 18526 hybrid_clock.cc:648] HybridClock initialized: now 1786307053291681 us; error 60 us; skew 500 ppm
I20260809 20:24:13.292068 18526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:13 dist-test-slave-rnrw krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307053, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:13.300654 18526 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:24:13.302279 18526 webserver.cc:460] Webserver started at http://127.12.78.62:36451/ using document root <none> and password file <none>
I20260809 20:24:13.302996 18526 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:13.303148 18526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:13.303478 18526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:13.306730 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "9407d45f2026461a915344aace349c7d"
format_stamp: "Formatted at 2026-08-09 20:24:13 on dist-test-slave-rnrw"
server_key: "27988baeb3a24aa451bbb069fe51b442"
server_key_iv: "3ff208c5fbe88ac1bcecec20a4b3e6e0"
server_key_version: "encryptionkey@0"
I20260809 20:24:13.307554 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "9407d45f2026461a915344aace349c7d"
format_stamp: "Formatted at 2026-08-09 20:24:13 on dist-test-slave-rnrw"
server_key: "27988baeb3a24aa451bbb069fe51b442"
server_key_iv: "3ff208c5fbe88ac1bcecec20a4b3e6e0"
server_key_version: "encryptionkey@0"
I20260809 20:24:13.312963 18526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 20:24:13.316388 18542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:13.317732 18526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:13.317907 18526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "9407d45f2026461a915344aace349c7d"
format_stamp: "Formatted at 2026-08-09 20:24:13 on dist-test-slave-rnrw"
server_key: "27988baeb3a24aa451bbb069fe51b442"
server_key_iv: "3ff208c5fbe88ac1bcecec20a4b3e6e0"
server_key_version: "encryptionkey@0"
I20260809 20:24:13.318110 18526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:13.344933 18526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:13.350391 18526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:13.350744 18526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:13.373512 18526 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:40509
I20260809 20:24:13.373579 18593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:40509 every 8 connection(s)
I20260809 20:24:13.375428 18526 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:24:13.380215 18594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:13.385383 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 18526
I20260809 20:24:13.385608 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:24:13.386319 12600 external_mini_cluster.cc:1428] Setting key 0db2a1849988608e7b919a43d47b9e68
I20260809 20:24:13.397783 18594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d: Bootstrap starting.
I20260809 20:24:13.400969 18594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d: Neither blocks nor log segments found. Creating new log.
Aug 09 20:24:13 dist-test-slave-rnrw krb5kdc[18512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:24:13.402273 18594 log.cc:824] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:13.405567 18594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d: No bootstrap required, opened a new log
I20260809 20:24:13.410964 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:13.388345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.12.78.62:40509)
0809 20:24:13.389432 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:13.389449 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:13.389510 (+ 61us) server_negotiation.cc:407] Connection header received
0809 20:24:13.391429 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:13.391453 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:13.391833 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:13.392239 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:13.393787 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:13.395207 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:13.396521 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:13.397611 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:13.397780 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:13.402530 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:13.402538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:13.402551 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:13.402590 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:13.406021 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:13.406872 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:13.406878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:13.406883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:13.407052 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:13.407571 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:13.407579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:13.407581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:13.408186 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:13.408455 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:13.409613 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0809 20:24:13.409933 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":657,"threads_started":1}
I20260809 20:24:13.414186 18594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9407d45f2026461a915344aace349c7d" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40509 } }
I20260809 20:24:13.414552 18594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:13.414664 18594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9407d45f2026461a915344aace349c7d, State: Initialized, Role: FOLLOWER
I20260809 20:24:13.415194 18594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9407d45f2026461a915344aace349c7d" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40509 } }
I20260809 20:24:13.415426 18594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:24:13.415547 18594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:24:13.415726 18594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:13.418042 18594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9407d45f2026461a915344aace349c7d" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40509 } }
I20260809 20:24:13.418529 18594 leader_election.cc:302] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9407d45f2026461a915344aace349c7d; no voters:
I20260809 20:24:13.419121 18594 leader_election.cc:288] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:24:13.419291 18599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:13.420087 18599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [term 1 LEADER]: Becoming Leader. State: Replica: 9407d45f2026461a915344aace349c7d, State: Running, Role: LEADER
I20260809 20:24:13.420724 18599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9407d45f2026461a915344aace349c7d" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40509 } }
I20260809 20:24:13.421370 18594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:13.423772 18601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9407d45f2026461a915344aace349c7d. Latest consensus state: current_term: 1 leader_uuid: "9407d45f2026461a915344aace349c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9407d45f2026461a915344aace349c7d" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40509 } } }
I20260809 20:24:13.424224 18601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:13.424873 18600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9407d45f2026461a915344aace349c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9407d45f2026461a915344aace349c7d" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40509 } } }
I20260809 20:24:13.425194 18600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:13.426571 18607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:13.432603 18607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:13.441370 18607 catalog_manager.cc:1347] Generated new cluster ID: b91331857f854c4fabe5f2cb105c069a
I20260809 20:24:13.441488 18607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:13.450821 18607 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:24:13.452286 18607 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:13.465342 18607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9407d45f2026461a915344aace349c7d: Generated new TSK 0
I20260809 20:24:13.466140 18607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:24:13.528299 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40509
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:13.867019 18620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:13.867309 18620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:13.867478 18620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:13.875327 18620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:13.875437 18620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:13.875643 18620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:24:13.885545 18620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:13.886863 18620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:13.888190 18620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:13.902045 18625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:13.902040 18628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:13.901974 18626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:13.902842 18620 server_base.cc:1034] running on GCE node
I20260809 20:24:13.903509 18620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:13.904445 18620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:13.905678 18620 hybrid_clock.cc:648] HybridClock initialized: now 1786307053905629 us; error 51 us; skew 500 ppm
I20260809 20:24:13.906002 18620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:13 dist-test-slave-rnrw krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307053, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:13.909473 18620 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:24:13.911221 18620 webserver.cc:460] Webserver started at http://127.12.78.1:45925/ using document root <none> and password file <none>
I20260809 20:24:13.911867 18620 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:13.911970 18620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:13.912233 18620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:13.915241 18620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "4c1ba3bd6eff47d3811a2ea74a8f3204"
format_stamp: "Formatted at 2026-08-09 20:24:13 on dist-test-slave-rnrw"
server_key: "715631032f845e13a1c0656e99e447b2"
server_key_iv: "ebc9e20d113c1cc0f076ebeaded3b9be"
server_key_version: "encryptionkey@0"
I20260809 20:24:13.915974 18620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "4c1ba3bd6eff47d3811a2ea74a8f3204"
format_stamp: "Formatted at 2026-08-09 20:24:13 on dist-test-slave-rnrw"
server_key: "715631032f845e13a1c0656e99e447b2"
server_key_iv: "ebc9e20d113c1cc0f076ebeaded3b9be"
server_key_version: "encryptionkey@0"
I20260809 20:24:13.921059 18620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 20:24:13.924726 18635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:13.926143 18620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:24:13.926316 18620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "4c1ba3bd6eff47d3811a2ea74a8f3204"
format_stamp: "Formatted at 2026-08-09 20:24:13 on dist-test-slave-rnrw"
server_key: "715631032f845e13a1c0656e99e447b2"
server_key_iv: "ebc9e20d113c1cc0f076ebeaded3b9be"
server_key_version: "encryptionkey@0"
I20260809 20:24:13.926508 18620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:13.951469 18620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:13.955734 18620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:13.956054 18620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:13.957149 18620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:13.958536 18620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:13.958621 18620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:13.958771 18620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:13.958828 18620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:13.993471 18620 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:34043
I20260809 20:24:13.993564 18751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:34043 every 8 connection(s)
I20260809 20:24:13.994833 18620 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:24:14.004347 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 18620
I20260809 20:24:14.004601 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:24:14.005254 12600 external_mini_cluster.cc:1428] Setting key 5b7c1b2905ae74398bea4f44b3ce6d98
Aug 09 20:24:14 dist-test-slave-rnrw krb5kdc[18512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307053, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:24:14.016433 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:13.997635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:36955 (local address 127.12.78.62:40509)
0809 20:24:13.998123 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:13.998128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:13.998884 (+ 756us) server_negotiation.cc:407] Connection header received
0809 20:24:14.000773 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:14.000780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:14.000872 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:14.000995 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:14.002497 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:14.003219 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:14.004925 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:14.005386 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:14.005561 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:14.009501 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:14.009511 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:14.009518 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:14.009565 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:14.013255 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:14.014064 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:14.014072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:14.014079 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:14.014238 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:14.014681 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:14.014688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:14.014693 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:14.015067 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:14.015225 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:14.015996 (+ 771us) server_negotiation.cc:299] Negotiation successful
0809 20:24:14.016180 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":266,"threads_started":1}
I20260809 20:24:14.018008 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:13.998013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40509 (local address 127.12.78.1:36955)
0809 20:24:13.998718 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:13.998764 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:14.000448 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:14.001234 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:14.001243 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:14.001748 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:14.002307 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:14.002325 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:14.003381 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:14.003385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:14.004782 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:14.004795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:14.005700 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:14.005708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:14.006017 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:14.006843 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:14.006864 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:14.009270 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:14.013426 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:14.013434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:14.013446 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:14.013921 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:14.014347 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:14.014350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:14.014352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:14.014531 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:14.015197 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:14.015203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:14.015512 (+ 309us) client_negotiation.cc:769] Sending connection context
0809 20:24:14.015890 (+ 378us) client_negotiation.cc:240] Negotiation successful
0809 20:24:14.016482 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":284,"threads_started":1}
I20260809 20:24:14.019636 18752 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40509
I20260809 20:24:14.019948 18752 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:14.020725 18752 heartbeater.cc:511] Master 127.12.78.62:40509 requested a full tablet report, sending...
I20260809 20:24:14.023269 18559 ts_manager.cc:194] Registered new tserver with Master: 4c1ba3bd6eff47d3811a2ea74a8f3204 (127.12.78.1:34043)
I20260809 20:24:14.025539 18559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:36955
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:24:14.059628 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40509
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:14.390746 18758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:14.391072 18758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:14.391233 18758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:14.398679 18758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:14.398876 18758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:14.399016 18758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:24:14.409325 18758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:14.410590 18758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:14.411881 18758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:14.425578 18763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:14.426011 18764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:14.425436 18766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:14.427258 18758 server_base.cc:1034] running on GCE node
I20260809 20:24:14.427915 18758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:14.428939 18758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:14.430243 18758 hybrid_clock.cc:648] HybridClock initialized: now 1786307054430144 us; error 72 us; skew 500 ppm
I20260809 20:24:14.430625 18758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:14 dist-test-slave-rnrw krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307054, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:14.434158 18758 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:24:14.435724 18758 webserver.cc:460] Webserver started at http://127.12.78.2:40485/ using document root <none> and password file <none>
I20260809 20:24:14.436440 18758 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:14.436611 18758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:14.436919 18758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:14.439900 18758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "b371c9c733df42d2adf92b68f260475b"
format_stamp: "Formatted at 2026-08-09 20:24:14 on dist-test-slave-rnrw"
server_key: "94d34bf4b2bafe72c54d06fb0971fcaa"
server_key_iv: "e3ffbd73c08806b6153a2268df7b1448"
server_key_version: "encryptionkey@0"
I20260809 20:24:14.440758 18758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "b371c9c733df42d2adf92b68f260475b"
format_stamp: "Formatted at 2026-08-09 20:24:14 on dist-test-slave-rnrw"
server_key: "94d34bf4b2bafe72c54d06fb0971fcaa"
server_key_iv: "e3ffbd73c08806b6153a2268df7b1448"
server_key_version: "encryptionkey@0"
I20260809 20:24:14.445878 18758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 20:24:14.449347 18773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:14.450809 18758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 20:24:14.451009 18758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "b371c9c733df42d2adf92b68f260475b"
format_stamp: "Formatted at 2026-08-09 20:24:14 on dist-test-slave-rnrw"
server_key: "94d34bf4b2bafe72c54d06fb0971fcaa"
server_key_iv: "e3ffbd73c08806b6153a2268df7b1448"
server_key_version: "encryptionkey@0"
I20260809 20:24:14.451223 18758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:14.466097 18758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:14.470357 18758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:14.470644 18758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:14.471576 18758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:14.472891 18758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:14.472995 18758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:14.473141 18758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:14.473198 18758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:14.506503 18758 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:36967
I20260809 20:24:14.506589 18888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:36967 every 8 connection(s)
I20260809 20:24:14.507925 18758 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:24:14.510931 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 18758
I20260809 20:24:14.511134 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:24:14.511574 12600 external_mini_cluster.cc:1428] Setting key bef961de9890d458ef672cd1235bd680
Aug 09 20:24:14 dist-test-slave-rnrw krb5kdc[18512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307054, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:24:14.530479 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:14.510392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:35205 (local address 127.12.78.62:40509)
0809 20:24:14.510648 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:14.510654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:14.511901 (+ 1247us) server_negotiation.cc:407] Connection header received
0809 20:24:14.513750 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:14.513756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:14.513836 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:14.513962 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:14.515355 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:14.516504 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:14.518014 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:14.518766 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:14.518934 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:14.522962 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:14.522971 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:14.522979 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:14.523033 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:14.526875 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:14.527612 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:14.527620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:14.527626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:14.527802 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:14.528245 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:14.528253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:14.528259 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:14.528749 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:14.528899 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:14.529686 (+ 787us) server_negotiation.cc:299] Negotiation successful
0809 20:24:14.529959 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 20:24:14.531327 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:14.510834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40509 (local address 127.12.78.2:35205)
0809 20:24:14.511725 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:14.511771 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:14.513503 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:14.514138 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:14.514146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:14.514641 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:14.515177 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:14.515193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:14.516675 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:14.516679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:14.517870 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:14.517882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:14.519074 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:14.519087 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:14.519378 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:14.520134 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:14.520159 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:14.522699 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:14.527022 (+ 4323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:14.527029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:14.527041 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:14.527457 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:14.527915 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:14.527918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:14.527920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:14.528120 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:14.528887 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:14.528893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:14.529252 (+ 359us) client_negotiation.cc:769] Sending connection context
0809 20:24:14.529557 (+ 305us) client_negotiation.cc:240] Negotiation successful
0809 20:24:14.529988 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":404,"threads_started":1}
I20260809 20:24:14.532892 18889 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40509
I20260809 20:24:14.533238 18889 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:14.534175 18889 heartbeater.cc:511] Master 127.12.78.62:40509 requested a full tablet report, sending...
I20260809 20:24:14.535807 18559 ts_manager.cc:194] Registered new tserver with Master: b371c9c733df42d2adf92b68f260475b (127.12.78.2:36967)
I20260809 20:24:14.536693 18559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:35205
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:24:14.568262 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40509
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:14.906401 18894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:14.906713 18894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:14.906872 18894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:14.914498 18894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:14.914625 18894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:14.914855 18894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:24:14.925199 18894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:14.926554 18894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:14.927918 18894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:14.941397 18900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:14.941356 18901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:14.941434 18903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:14.942408 18894 server_base.cc:1034] running on GCE node
I20260809 20:24:14.942951 18894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:14.943878 18894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:14.945072 18894 hybrid_clock.cc:648] HybridClock initialized: now 1786307054945041 us; error 43 us; skew 500 ppm
I20260809 20:24:14.945386 18894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:14 dist-test-slave-rnrw krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307054, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:14.948518 18894 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:24:14.950021 18894 webserver.cc:460] Webserver started at http://127.12.78.3:36001/ using document root <none> and password file <none>
I20260809 20:24:14.950706 18894 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:14.950841 18894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:14.951179 18894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:14.954150 18894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "34475e3bdf434410ac0894a5dea7409e"
format_stamp: "Formatted at 2026-08-09 20:24:14 on dist-test-slave-rnrw"
server_key: "f04df0cf9c58a790372238a778612f13"
server_key_iv: "c2db21c5322c6471d00514825c3054aa"
server_key_version: "encryptionkey@0"
I20260809 20:24:14.955106 18894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "34475e3bdf434410ac0894a5dea7409e"
format_stamp: "Formatted at 2026-08-09 20:24:14 on dist-test-slave-rnrw"
server_key: "f04df0cf9c58a790372238a778612f13"
server_key_iv: "c2db21c5322c6471d00514825c3054aa"
server_key_version: "encryptionkey@0"
I20260809 20:24:14.960703 18894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 20:24:14.964133 18910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:14.965571 18894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:14.965752 18894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "34475e3bdf434410ac0894a5dea7409e"
format_stamp: "Formatted at 2026-08-09 20:24:14 on dist-test-slave-rnrw"
server_key: "f04df0cf9c58a790372238a778612f13"
server_key_iv: "c2db21c5322c6471d00514825c3054aa"
server_key_version: "encryptionkey@0"
I20260809 20:24:14.965965 18894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:14.991132 18894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:14.996935 18894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:14.997349 18894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:14.998594 18894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:15.000422 18894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:15.000517 18894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:15.000679 18894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:15.000761 18894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:15.028764 18752 heartbeater.cc:503] Master 127.12.78.62:40509 was elected leader, sending a full tablet report...
I20260809 20:24:15.035542 18894 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:34455
I20260809 20:24:15.035586 19025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:34455 every 8 connection(s)
I20260809 20:24:15.036917 18894 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:24:15.041836 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 18894
I20260809 20:24:15.042050 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:24:15.042472 12600 external_mini_cluster.cc:1428] Setting key da67dae5b6728dba1d08128d524b0539
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[18512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307054, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:24:15.057211 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:15.039452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:38741 (local address 127.12.78.62:40509)
0809 20:24:15.039901 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:15.039908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:15.040921 (+ 1013us) server_negotiation.cc:407] Connection header received
0809 20:24:15.042805 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:15.042811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:15.042906 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:15.043041 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:15.044759 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.045886 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.047351 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.047770 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.047864 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:15.051240 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:15.051245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:15.051249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:15.051282 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:15.054072 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.054747 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.054751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.054754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.054886 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.055343 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.055347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.055349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.055613 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:15.055728 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:15.056464 (+ 736us) server_negotiation.cc:299] Negotiation successful
0809 20:24:15.056813 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":207,"threads_started":1}
I20260809 20:24:15.058135 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:15.040027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40509 (local address 127.12.78.3:38741)
0809 20:24:15.040771 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:15.040815 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:15.042504 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:15.043300 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:15.043309 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:15.043902 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:15.044587 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:15.044624 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.046028 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.046032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:15.047216 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:15.047228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.047950 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.047954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:15.048140 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:15.048855 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:15.048876 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:15.051044 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:15.054208 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:15.054214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:15.054225 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:15.054637 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:15.055025 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:15.055032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:15.055036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:15.055222 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:15.055736 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:15.055742 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:15.056017 (+ 275us) client_negotiation.cc:769] Sending connection context
0809 20:24:15.056332 (+ 315us) client_negotiation.cc:240] Negotiation successful
0809 20:24:15.056875 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":276,"threads_started":1}
I20260809 20:24:15.059368 19026 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40509
I20260809 20:24:15.059636 19026 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:15.060259 19026 heartbeater.cc:511] Master 127.12.78.62:40509 requested a full tablet report, sending...
I20260809 20:24:15.061625 18559 ts_manager.cc:194] Registered new tserver with Master: 34475e3bdf434410ac0894a5dea7409e (127.12.78.3:34455)
I20260809 20:24:15.062356 18559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:38741
I20260809 20:24:15.070379 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/test-admin.keytab.
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:15.101994 12600 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 20:24:15.107805 19034 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:15.104949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49590 (local address 127.12.78.1:34043)
0809 20:24:15.105642 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:15.105649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:15.105675 (+ 26us) server_negotiation.cc:407] Connection header received
0809 20:24:15.105738 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:15.105744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:15.105940 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:15.106082 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:15.107295 (+ 1213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49590: BlockingRecv error: recv got EOF from 127.0.0.1:49590 (error 108)
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":386,"threads_started":1}
W20260809 20:24:15.108091 12600 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/krb5cc' of type FILE
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:15.109903 12600 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:15.111334 12600 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[18512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:24:15.127697 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:15.113162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.12.78.1:34043)
0809 20:24:15.113368 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:15.113373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:15.113554 (+ 181us) server_negotiation.cc:407] Connection header received
0809 20:24:15.113757 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:15.113761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:15.113818 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:15.113920 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:15.114932 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.115880 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.117167 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.117745 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.117886 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:15.122113 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:15.122127 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:15.122152 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:15.122189 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:15.125010 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.125671 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.125679 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.125681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.125802 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.126163 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.126169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.126172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.126521 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:15.126827 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:15.127148 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 20:24:15.127396 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260809 20:24:15.133546 19034 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:15.131917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49598 (local address 127.12.78.1:34043)
0809 20:24:15.132152 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:15.132157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:15.132344 (+ 187us) server_negotiation.cc:407] Connection header received
0809 20:24:15.132447 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:15.132450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:15.132509 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:15.132636 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:15.133310 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49598: BlockingRecv error: recv got EOF from 127.0.0.1:49598 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260809 20:24:15.133661 12600 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/krb5cc' of type FILE
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:15.135195 12600 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:15.136706 12600 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[18512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:24:15.150799 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:15.137952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.12.78.1:34043)
0809 20:24:15.138153 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:15.138160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:15.138435 (+ 275us) server_negotiation.cc:407] Connection header received
0809 20:24:15.138657 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:15.138662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:15.138723 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:15.138820 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:15.139701 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.140383 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.141628 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.142071 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.142201 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:15.145687 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:15.145691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:15.145695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:15.145735 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:15.148258 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.149005 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.149017 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.149021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.149156 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.149537 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.149546 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.149548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.149806 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:15.150090 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:15.150333 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 20:24:15.150533 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260809 20:24:15.154938 19034 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:15.153395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.12.78.1:34043)
0809 20:24:15.153598 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:15.153603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:15.153689 (+ 86us) server_negotiation.cc:407] Connection header received
0809 20:24:15.153801 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:15.153805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:15.153869 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:15.153944 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:15.154676 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49606: BlockingRecv error: recv got EOF from 127.0.0.1:49606 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260809 20:24:15.155004 12600 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786307001768029-12600-0/krb5kdc/krb5cc' of type FILE
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:15.156586 12600 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:15.157987 12600 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[18512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:24:15.172184 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:15.159317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49614 (local address 127.12.78.1:34043)
0809 20:24:15.159464 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:15.159468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:15.159857 (+ 389us) server_negotiation.cc:407] Connection header received
0809 20:24:15.159972 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:15.159977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:15.160031 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:15.160107 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:15.161056 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.161755 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.162947 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.163381 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.163524 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:15.166827 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:15.166832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:15.166835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:15.166864 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:15.169563 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.170220 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.170224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.170226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.170377 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.170791 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.170795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.170798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.171071 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:15.171306 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:15.171579 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 20:24:15.171867 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 20:24:15.173959 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 18620
I20260809 20:24:15.185359 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:15.174764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49622 (local address 127.12.78.1:34043)
0809 20:24:15.174910 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:15.174914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:15.175325 (+ 411us) server_negotiation.cc:407] Connection header received
0809 20:24:15.175443 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:15.175447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:15.175496 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:15.175572 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:15.176470 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.177220 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.178377 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.178785 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.178872 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:15.180821 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:15.180826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:15.180829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:15.180856 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:15.183168 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.183725 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.183728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.183731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.183858 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.184167 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.184170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.184172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.184401 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:15.184668 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:15.184943 (+ 275us) server_negotiation.cc:299] Negotiation successful
0809 20:24:15.185100 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 20:24:15.185945 18747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:15.273159 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 18620
I20260809 20:24:15.293164 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 18758
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[18512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:24:15.309201 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:15.294086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.12.78.2:36967)
0809 20:24:15.294649 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:15.294656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:15.294683 (+ 27us) server_negotiation.cc:407] Connection header received
0809 20:24:15.294742 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:15.294749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:15.294959 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:15.295097 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:15.296080 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.297327 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.298573 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.299206 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.299324 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:15.303401 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:15.303413 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:15.303429 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:15.303469 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:15.306368 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.307013 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.307020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.307022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.307151 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.307483 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.307486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.307488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.307827 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:15.308166 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:15.308443 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 20:24:15.308754 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":293,"threads_started":1}
I20260809 20:24:15.310582 18884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:15.395681 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 18758
I20260809 20:24:15.414099 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 18894
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[18512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:24:15.431663 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:15.415226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.12.78.3:34455)
0809 20:24:15.415923 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:15.415930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:15.415981 (+ 51us) server_negotiation.cc:407] Connection header received
0809 20:24:15.416045 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:15.416051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:15.416265 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:15.416405 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:15.417541 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.418708 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.419949 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:15.420690 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:15.420820 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:15.425423 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:15.425441 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:15.425457 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:15.425547 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:15.428815 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.429620 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.429628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.429630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.429767 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:15.430131 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:15.430135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:15.430137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:15.430537 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:15.430780 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:15.431028 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 20:24:15.431304 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":364,"threads_started":1}
I20260809 20:24:15.433024 19021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:15.516937 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 18894
I20260809 20:24:15.535085 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 18526
I20260809 20:24:15.536336 18589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:15.641295 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 18526
2026-08-09T20:24:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4094 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[19054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[19054](info): set up 2 sockets
Aug 09 20:24:15 dist-test-slave-rnrw krb5kdc[19054](info): commencing operation
krb5kdc: starting...
W20260809 20:24:16.810472 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:24:16 dist-test-slave-rnrw krb5kdc[19054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:24:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:24:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.62.keytab.
Entry for principal oryx/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.62.keytab.
I20260809 20:24:17.056136 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:39677
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.62.keytab
--principal=oryx/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:39677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:17.400877 19068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:17.401233 19068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:17.401388 19068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:17.408849 19068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:17.409020 19068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:17.409129 19068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:17.409232 19068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:17.409312 19068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:24:17.419561 19068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:39677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.62.keytab
--principal=oryx/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:39677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:17.421087 19068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:17.422469 19068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:17.434947 19074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:17.434861 19077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:17.434890 19075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:17.435694 19068 server_base.cc:1034] running on GCE node
I20260809 20:24:17.436437 19068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:17.437889 19068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:17.439144 19068 hybrid_clock.cc:648] HybridClock initialized: now 1786307057439091 us; error 59 us; skew 500 ppm
I20260809 20:24:17.439502 19068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:17 dist-test-slave-rnrw krb5kdc[19054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307057, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:17.442572 19068 init.cc:377] Logged in from keytab as oryx/127.12.78.62@KRBTEST.COM (short username oryx)
I20260809 20:24:17.444001 19068 webserver.cc:460] Webserver started at http://127.12.78.62:34095/ using document root <none> and password file <none>
I20260809 20:24:17.444793 19068 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:17.444941 19068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:17.445251 19068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:17.448194 19068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "ccd8c1717f7b4ce8b2489ede55f20522"
format_stamp: "Formatted at 2026-08-09 20:24:17 on dist-test-slave-rnrw"
server_key: "39dc1268868fd516ed565498dde7fa6e"
server_key_iv: "1f75b169f2eeaf2bcc3f71b93bacca54"
server_key_version: "encryptionkey@0"
I20260809 20:24:17.449023 19068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "ccd8c1717f7b4ce8b2489ede55f20522"
format_stamp: "Formatted at 2026-08-09 20:24:17 on dist-test-slave-rnrw"
server_key: "39dc1268868fd516ed565498dde7fa6e"
server_key_iv: "1f75b169f2eeaf2bcc3f71b93bacca54"
server_key_version: "encryptionkey@0"
I20260809 20:24:17.453964 19068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 20:24:17.457299 19084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:17.458626 19068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:24:17.458767 19068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "ccd8c1717f7b4ce8b2489ede55f20522"
format_stamp: "Formatted at 2026-08-09 20:24:17 on dist-test-slave-rnrw"
server_key: "39dc1268868fd516ed565498dde7fa6e"
server_key_iv: "1f75b169f2eeaf2bcc3f71b93bacca54"
server_key_version: "encryptionkey@0"
I20260809 20:24:17.458923 19068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:17.505319 19068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:17.509689 19068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:17.509941 19068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:17.529449 19068 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:39677
I20260809 20:24:17.529482 19136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:39677 every 8 connection(s)
I20260809 20:24:17.530882 19068 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:24:17.531522 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 19068
I20260809 20:24:17.531824 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:24:17.532675 12600 external_mini_cluster.cc:1428] Setting key 13f63842aca5ff3cc77c7eb2f7cdd044
I20260809 20:24:17.535568 19137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 20:24:17 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
I20260809 20:24:17.549568 19137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522: Bootstrap starting.
I20260809 20:24:17.554037 19137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:17.555397 19137 log.cc:824] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:17.558435 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:17.534888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.12.78.62:39677)
0809 20:24:17.535865 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:17.535877 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:17.535929 (+ 52us) server_negotiation.cc:407] Connection header received
0809 20:24:17.537762 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:17.537784 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:17.538215 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:17.538591 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:17.540517 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:17.541994 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:17.543766 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:17.544431 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:17.544633 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:17.549437 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:17.549449 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:17.549468 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:17.549515 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:17.553571 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:17.554333 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:17.554348 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:17.554360 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:17.554605 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:17.555030 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:17.555038 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:17.555042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:17.555798 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:17.556180 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:17.556673 (+ 493us) server_negotiation.cc:299] Negotiation successful
0809 20:24:17.557091 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":446,"threads_started":1}
I20260809 20:24:17.559092 19137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522: No bootstrap required, opened a new log
I20260809 20:24:17.568112 19137 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } }
I20260809 20:24:17.568483 19137 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:17.568614 19137 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd8c1717f7b4ce8b2489ede55f20522, State: Initialized, Role: FOLLOWER
I20260809 20:24:17.569129 19137 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } }
I20260809 20:24:17.569432 19137 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:24:17.569645 19137 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:24:17.569844 19137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:17.571992 19137 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } }
I20260809 20:24:17.572492 19137 leader_election.cc:302] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccd8c1717f7b4ce8b2489ede55f20522; no voters:
I20260809 20:24:17.573168 19137 leader_election.cc:288] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:24:17.573315 19142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:17.574024 19142 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [term 1 LEADER]: Becoming Leader. State: Replica: ccd8c1717f7b4ce8b2489ede55f20522, State: Running, Role: LEADER
I20260809 20:24:17.574586 19142 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } }
I20260809 20:24:17.575271 19137 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:17.577945 19144 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccd8c1717f7b4ce8b2489ede55f20522. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } }
I20260809 20:24:17.578390 19144 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:17.579252 19143 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } }
I20260809 20:24:17.579605 19143 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:17.580299 19151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:17.585427 19151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:17.594259 19151 catalog_manager.cc:1347] Generated new cluster ID: 958da1061cb0417788c3a84c366f3f25
I20260809 20:24:17.594388 19151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:17.626014 19151 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:24:17.627919 19151 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:17.635241 19151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522: Generated new TSK 0
I20260809 20:24:17.636230 19151 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab.
Entry for principal oryx/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab.
I20260809 20:24:17.706621 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:39677
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab
--principal=oryx/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:18.046969 19164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:18.047248 19164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:18.047343 19164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:18.054642 19164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:18.054751 19164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:18.054943 19164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:24:18.064827 19164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab
--principal=oryx/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:18.066139 19164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:18.067494 19164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:18.081003 19170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:18.081080 19171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:18.080981 19173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:18.081889 19164 server_base.cc:1034] running on GCE node
I20260809 20:24:18.082544 19164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:18.083513 19164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:18.084713 19164 hybrid_clock.cc:648] HybridClock initialized: now 1786307058084645 us; error 64 us; skew 500 ppm
I20260809 20:24:18.085029 19164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:18 dist-test-slave-rnrw krb5kdc[19054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307058, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:18.088266 19164 init.cc:377] Logged in from keytab as oryx/127.12.78.1@KRBTEST.COM (short username oryx)
I20260809 20:24:18.089830 19164 webserver.cc:460] Webserver started at http://127.12.78.1:35835/ using document root <none> and password file <none>
I20260809 20:24:18.090535 19164 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:18.090690 19164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:18.091050 19164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:18.094026 19164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "24068fc4286f45a4bfe48c531efac812"
format_stamp: "Formatted at 2026-08-09 20:24:18 on dist-test-slave-rnrw"
server_key: "c6389f4303735fb0a9de1ca6ab70fbe4"
server_key_iv: "9648b249794fc11981a1804be59ec1c1"
server_key_version: "encryptionkey@0"
I20260809 20:24:18.094810 19164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "24068fc4286f45a4bfe48c531efac812"
format_stamp: "Formatted at 2026-08-09 20:24:18 on dist-test-slave-rnrw"
server_key: "c6389f4303735fb0a9de1ca6ab70fbe4"
server_key_iv: "9648b249794fc11981a1804be59ec1c1"
server_key_version: "encryptionkey@0"
I20260809 20:24:18.100101 19164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 20:24:18.103623 19180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:18.105050 19164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 20:24:18.105229 19164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "24068fc4286f45a4bfe48c531efac812"
format_stamp: "Formatted at 2026-08-09 20:24:18 on dist-test-slave-rnrw"
server_key: "c6389f4303735fb0a9de1ca6ab70fbe4"
server_key_iv: "9648b249794fc11981a1804be59ec1c1"
server_key_version: "encryptionkey@0"
I20260809 20:24:18.105435 19164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:18.128109 19164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:18.132174 19164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:18.132498 19164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:18.133500 19164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:18.134948 19164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:18.135063 19164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:18.135212 19164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:18.135282 19164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:18.169142 19164 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:45283
I20260809 20:24:18.169205 19301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:45283 every 8 connection(s)
I20260809 20:24:18.170493 19164 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:24:18.171223 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 19164
I20260809 20:24:18.171499 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:24:18.172290 12600 external_mini_cluster.cc:1428] Setting key ec12b5692959759a83f4368c815ad1ce
Aug 09 20:24:18 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307058, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
I20260809 20:24:18.194329 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:18.173551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:46793 (local address 127.12.78.62:39677)
0809 20:24:18.173999 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:18.174003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:18.175200 (+ 1197us) server_negotiation.cc:407] Connection header received
0809 20:24:18.178171 (+ 2971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:18.178175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:18.178242 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:18.178335 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:18.180455 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:18.181243 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:18.183775 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:18.184499 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:18.184689 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:18.188216 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:18.188221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:18.188224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:18.188257 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:18.191065 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:18.191873 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:18.191884 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:18.191887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:18.192016 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:18.192481 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:18.192491 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:18.192493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:18.192816 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:18.192985 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:18.193702 (+ 717us) server_negotiation.cc:299] Negotiation successful
0809 20:24:18.193941 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":249,"threads_started":1}
I20260809 20:24:18.195330 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:18.174028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.1:46793)
0809 20:24:18.174997 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:18.175050 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:18.177873 (+ 2823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:18.178612 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:18.178629 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:18.179440 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:18.180270 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:18.180299 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:18.181407 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:18.181415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:18.183580 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:18.183598 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:18.184790 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:18.184796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:18.184998 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:18.185691 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:18.185713 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:18.187980 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:18.191222 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:18.191232 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:18.191250 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:18.191735 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:18.192139 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:18.192144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:18.192147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:18.192350 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:18.192971 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:18.192979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:18.193318 (+ 339us) client_negotiation.cc:769] Sending connection context
0809 20:24:18.193601 (+ 283us) client_negotiation.cc:240] Negotiation successful
0809 20:24:18.194157 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":418,"threads_started":1}
WARNING: no policy specified for oryx/127.12.78.2@KRBTEST.COM; defaulting to no policy
I20260809 20:24:18.197166 19302 heartbeater.cc:347] Connected to a master server at 127.12.78.62:39677
I20260809 20:24:18.197494 19302 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:18.198311 19302 heartbeater.cc:511] Master 127.12.78.62:39677 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.78.2@KRBTEST.COM" created.
I20260809 20:24:18.200708 19102 ts_manager.cc:194] Registered new tserver with Master: 24068fc4286f45a4bfe48c531efac812 (127.12.78.1:45283)
I20260809 20:24:18.202168 19102 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:46793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab.
Entry for principal oryx/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab.
I20260809 20:24:18.227532 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:39677
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab
--principal=oryx/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:18.568804 19309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:18.569095 19309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:18.569195 19309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:18.576454 19309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:18.576629 19309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:18.576756 19309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:24:18.586586 19309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab
--principal=oryx/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:18.587929 19309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:18.589224 19309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:18.602142 19314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:18.602192 19317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:18.602192 19315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:18.603109 19309 server_base.cc:1034] running on GCE node
I20260809 20:24:18.603668 19309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:18.604583 19309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:18.605811 19309 hybrid_clock.cc:648] HybridClock initialized: now 1786307058605747 us; error 45 us; skew 500 ppm
I20260809 20:24:18.606140 19309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:18 dist-test-slave-rnrw krb5kdc[19054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307058, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:18.614099 19309 init.cc:377] Logged in from keytab as oryx/127.12.78.2@KRBTEST.COM (short username oryx)
I20260809 20:24:18.615804 19309 webserver.cc:460] Webserver started at http://127.12.78.2:38533/ using document root <none> and password file <none>
I20260809 20:24:18.616657 19309 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:18.616799 19309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:18.617122 19309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:18.620023 19309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "fc594aa5090a47c5b511a917b5c7b24b"
format_stamp: "Formatted at 2026-08-09 20:24:18 on dist-test-slave-rnrw"
server_key: "538af637c5df1bff016c3ef1abff3927"
server_key_iv: "8b8d51619df12803090001b798a4185e"
server_key_version: "encryptionkey@0"
I20260809 20:24:18.620896 19309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "fc594aa5090a47c5b511a917b5c7b24b"
format_stamp: "Formatted at 2026-08-09 20:24:18 on dist-test-slave-rnrw"
server_key: "538af637c5df1bff016c3ef1abff3927"
server_key_iv: "8b8d51619df12803090001b798a4185e"
server_key_version: "encryptionkey@0"
I20260809 20:24:18.625982 19309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 20:24:18.629575 19324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:18.630925 19309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 20:24:18.631100 19309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "fc594aa5090a47c5b511a917b5c7b24b"
format_stamp: "Formatted at 2026-08-09 20:24:18 on dist-test-slave-rnrw"
server_key: "538af637c5df1bff016c3ef1abff3927"
server_key_iv: "8b8d51619df12803090001b798a4185e"
server_key_version: "encryptionkey@0"
I20260809 20:24:18.631305 19309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:18.689563 19309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:18.694043 19309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:18.694350 19309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:18.695326 19309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:18.696875 19309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:18.696978 19309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:18.697131 19309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:18.697204 19309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:18.732679 19309 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:37183
I20260809 20:24:18.732712 19441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:37183 every 8 connection(s)
I20260809 20:24:18.733992 19309 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:24:18.742224 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 19309
I20260809 20:24:18.742444 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:24:18.742894 12600 external_mini_cluster.cc:1428] Setting key 79a0dc1deff531d52b4614db81d5130d
Aug 09 20:24:18 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307058, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
I20260809 20:24:18.754532 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:18.736838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:59911 (local address 127.12.78.62:39677)
0809 20:24:18.737340 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:18.737345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:18.738186 (+ 841us) server_negotiation.cc:407] Connection header received
0809 20:24:18.739959 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:18.739964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:18.740059 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:18.740177 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:18.741719 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:18.742552 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:18.744306 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:18.744977 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:18.745114 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:18.748845 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:18.748851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:18.748854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:18.748886 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:18.751374 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:18.752100 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:18.752105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:18.752107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:18.752240 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:18.752751 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:18.752755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:18.752757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:18.753022 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:18.753219 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:18.753913 (+ 694us) server_negotiation.cc:299] Negotiation successful
0809 20:24:18.754155 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":291,"threads_started":1}
I20260809 20:24:18.755489 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:18.737305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.2:59911)
0809 20:24:18.738007 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:18.738051 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:18.739689 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:18.740417 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:18.740425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:18.741005 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:18.741544 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:18.741560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:18.742719 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:18.742723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:18.744131 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:18.744143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:18.745228 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:18.745236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:18.745462 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:18.746159 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:18.746180 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:18.748602 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:18.751517 (+ 2915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:18.751526 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:18.751541 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:18.751962 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:18.752355 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:18.752361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:18.752364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:18.752608 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:18.753203 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:18.753210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:18.753541 (+ 331us) client_negotiation.cc:769] Sending connection context
0809 20:24:18.753813 (+ 272us) client_negotiation.cc:240] Negotiation successful
0809 20:24:18.754342 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":290,"threads_started":1}
I20260809 20:24:18.756973 19442 heartbeater.cc:347] Connected to a master server at 127.12.78.62:39677
I20260809 20:24:18.757308 19442 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:18.758188 19442 heartbeater.cc:511] Master 127.12.78.62:39677 requested a full tablet report, sending...
I20260809 20:24:18.759841 19102 ts_manager.cc:194] Registered new tserver with Master: fc594aa5090a47c5b511a917b5c7b24b (127.12.78.2:37183)
I20260809 20:24:18.760721 19102 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:59911
WARNING: no policy specified for oryx/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab.
Entry for principal oryx/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab.
I20260809 20:24:18.797887 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:39677
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab
--principal=oryx/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:19.134542 19448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:19.134824 19448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:19.135035 19448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:19.142107 19448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:19.142235 19448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:19.142426 19448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:24:19.151815 19448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab
--principal=oryx/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:19.153108 19448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:19.154476 19448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:19.167553 19454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:19.167505 19457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:19.167600 19455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:19.168637 19448 server_base.cc:1034] running on GCE node
I20260809 20:24:19.169322 19448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:19.170326 19448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:19.171554 19448 hybrid_clock.cc:648] HybridClock initialized: now 1786307059171497 us; error 51 us; skew 500 ppm
I20260809 20:24:19.171877 19448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:19 dist-test-slave-rnrw krb5kdc[19054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307059, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:19.175097 19448 init.cc:377] Logged in from keytab as oryx/127.12.78.3@KRBTEST.COM (short username oryx)
I20260809 20:24:19.176934 19448 webserver.cc:460] Webserver started at http://127.12.78.3:41219/ using document root <none> and password file <none>
I20260809 20:24:19.177639 19448 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:19.177793 19448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:19.178164 19448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:19.181267 19448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "5d4d517e110a40bb9f5f3caae6e3e607"
format_stamp: "Formatted at 2026-08-09 20:24:19 on dist-test-slave-rnrw"
server_key: "80c87741cd110bd382ca6e0bef85921f"
server_key_iv: "5ecc0f18ad849385e5f38f40a6f252e7"
server_key_version: "encryptionkey@0"
I20260809 20:24:19.182052 19448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "5d4d517e110a40bb9f5f3caae6e3e607"
format_stamp: "Formatted at 2026-08-09 20:24:19 on dist-test-slave-rnrw"
server_key: "80c87741cd110bd382ca6e0bef85921f"
server_key_iv: "5ecc0f18ad849385e5f38f40a6f252e7"
server_key_version: "encryptionkey@0"
I20260809 20:24:19.187279 19448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 20:24:19.190793 19464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:19.192101 19448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 20:24:19.192296 19448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "5d4d517e110a40bb9f5f3caae6e3e607"
format_stamp: "Formatted at 2026-08-09 20:24:19 on dist-test-slave-rnrw"
server_key: "80c87741cd110bd382ca6e0bef85921f"
server_key_iv: "5ecc0f18ad849385e5f38f40a6f252e7"
server_key_version: "encryptionkey@0"
I20260809 20:24:19.192497 19448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:19.205144 19302 heartbeater.cc:503] Master 127.12.78.62:39677 was elected leader, sending a full tablet report...
I20260809 20:24:19.229419 19448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:19.233557 19448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:19.233808 19448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:19.234913 19448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:19.236367 19448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:19.236451 19448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:19.236536 19448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:19.236644 19448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 20:24:19.271430 19448 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:43915
I20260809 20:24:19.271538 19580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:43915 every 8 connection(s)
I20260809 20:24:19.272842 19448 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:24:19.280462 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 19448
I20260809 20:24:19.280719 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:24:19.281266 12600 external_mini_cluster.cc:1428] Setting key aae25d6be73b21f9a8e04421c5afb835
Aug 09 20:24:19 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307059, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
I20260809 20:24:19.294665 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:19.275532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:32971 (local address 127.12.78.62:39677)
0809 20:24:19.276050 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:19.276064 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:19.277073 (+ 1009us) server_negotiation.cc:407] Connection header received
0809 20:24:19.279427 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:19.279433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:19.279530 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:19.279649 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:19.281937 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.282956 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.284857 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.285245 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.285378 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:19.288970 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:19.288975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:19.288978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:19.289011 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:19.291590 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:19.292325 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:19.292330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:19.292332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:19.292451 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:19.292920 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:19.292925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:19.292930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:19.293196 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:19.293372 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:19.294037 (+ 665us) server_negotiation.cc:299] Negotiation successful
0809 20:24:19.294261 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":221,"threads_started":1}
I20260809 20:24:19.295866 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:19.276108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.3:32971)
0809 20:24:19.276899 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:19.276938 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:19.279093 (+ 2155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:19.279985 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:19.279997 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:19.280911 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:19.281656 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:19.281676 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.283115 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.283121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:19.284708 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:19.284721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.285479 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.285488 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:19.285730 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:19.286330 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:19.286353 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:19.288747 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:19.291732 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:19.291741 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:19.291757 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:19.292188 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:19.292598 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:19.292603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:19.292606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:19.292797 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:19.293335 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:19.293343 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:19.293672 (+ 329us) client_negotiation.cc:769] Sending connection context
0809 20:24:19.293930 (+ 258us) client_negotiation.cc:240] Negotiation successful
0809 20:24:19.294486 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":307,"threads_started":1}
I20260809 20:24:19.297361 19581 heartbeater.cc:347] Connected to a master server at 127.12.78.62:39677
I20260809 20:24:19.297673 19581 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:19.298410 19581 heartbeater.cc:511] Master 127.12.78.62:39677 requested a full tablet report, sending...
I20260809 20:24:19.299902 19101 ts_manager.cc:194] Registered new tserver with Master: 5d4d517e110a40bb9f5f3caae6e3e607 (127.12.78.3:43915)
I20260809 20:24:19.300827 19101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:32971
I20260809 20:24:19.309273 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.78.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.78.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.78.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.61.keytab.
Entry for principal oryx/127.12.78.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.61.keytab.
I20260809 20:24:19.364190 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.61:39915
--webserver_interface=127.12.78.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.61.keytab
--principal=oryx/127.12.78.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:39677,127.12.78.61:39915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:19.707466 19587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:19.707777 19587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:19.707940 19587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:19.715503 19587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:19.715618 19587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:19.715747 19587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:19.715837 19587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:19.715902 19587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:24:19.727015 19587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:39677,127.12.78.61:39915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.61.keytab
--principal=oryx/127.12.78.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.61:39915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.78.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
W20260809 20:24:19.727427 19587 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.78.62:39677,127.12.78.61:39915'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 20:24:19.728617 19587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:19.730026 19587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:19.743424 19592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:19.743467 19595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:19.743657 19593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:19.744400 19587 server_base.cc:1034] running on GCE node
I20260809 20:24:19.745232 19587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:19.746688 19587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:19.747915 19587 hybrid_clock.cc:648] HybridClock initialized: now 1786307059747854 us; error 37 us; skew 500 ppm
I20260809 20:24:19.748329 19587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:19 dist-test-slave-rnrw krb5kdc[19054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307059, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:19.751542 19587 init.cc:377] Logged in from keytab as oryx/127.12.78.61@KRBTEST.COM (short username oryx)
I20260809 20:24:19.753079 19587 webserver.cc:460] Webserver started at http://127.12.78.61:45855/ using document root <none> and password file <none>
I20260809 20:24:19.753773 19587 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:19.753906 19587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:19.754226 19587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:19.757287 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/data/instance:
uuid: "45dcfa89acf6426c930dddd0e37f1254"
format_stamp: "Formatted at 2026-08-09 20:24:19 on dist-test-slave-rnrw"
server_key: "a298ba5291ecc2518838aefb66cb0847"
server_key_iv: "7572b90b71dc89b45bbe4b1d61a09285"
server_key_version: "encryptionkey@0"
I20260809 20:24:19.758090 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/wal/instance:
uuid: "45dcfa89acf6426c930dddd0e37f1254"
format_stamp: "Formatted at 2026-08-09 20:24:19 on dist-test-slave-rnrw"
server_key: "a298ba5291ecc2518838aefb66cb0847"
server_key_iv: "7572b90b71dc89b45bbe4b1d61a09285"
server_key_version: "encryptionkey@0"
I20260809 20:24:19.763760 19587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:24:19.764110 19442 heartbeater.cc:503] Master 127.12.78.62:39677 was elected leader, sending a full tablet report...
I20260809 20:24:19.768026 19603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:19.769483 19587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:19.769668 19587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/wal
uuid: "45dcfa89acf6426c930dddd0e37f1254"
format_stamp: "Formatted at 2026-08-09 20:24:19 on dist-test-slave-rnrw"
server_key: "a298ba5291ecc2518838aefb66cb0847"
server_key_iv: "7572b90b71dc89b45bbe4b1d61a09285"
server_key_version: "encryptionkey@0"
I20260809 20:24:19.769901 19587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:19.794060 19587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:19.800484 19587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:19.800827 19587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:19.820264 19587 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.61:39915
I20260809 20:24:19.820268 19656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.61:39915 every 8 connection(s)
I20260809 20:24:19.822182 19587 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/data/info.pb
I20260809 20:24:19.826963 19657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:19.827965 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 19587
I20260809 20:24:19.841779 19657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:24:19 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.78.61@KRBTEST.COM
I20260809 20:24:19.865898 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:19.831125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.12.78.61:39915)
0809 20:24:19.832155 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:19.832178 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:19.837397 (+ 5219us) server_negotiation.cc:407] Connection header received
0809 20:24:19.839210 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:19.839240 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:19.839809 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:19.840238 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:19.842412 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.844318 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.846589 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.847984 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.848206 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:19.855602 (+ 7396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:19.855618 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:19.855644 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:19.855711 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:19.861011 (+ 5300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:19.862011 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:19.862021 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:19.862029 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:19.862268 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:19.862779 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:19.862784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:19.862789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:19.863391 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:19.863744 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:19.864227 (+ 483us) server_negotiation.cc:299] Negotiation successful
0809 20:24:19.864856 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":410,"threads_started":1}
Aug 09 20:24:19 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307059, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.61@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
I20260809 20:24:19.878656 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:19.844864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.12.78.62:39677)
0809 20:24:19.860095 (+ 15231us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:19.860101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:19.860127 (+ 26us) server_negotiation.cc:407] Connection header received
0809 20:24:19.860477 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:19.860484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:19.860624 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:19.860757 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:19.862651 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.864215 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.866823 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.867390 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.867566 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:19.871587 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:19.871592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:19.871595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:19.871666 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:19.875330 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:19.876126 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:19.876134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:19.876147 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:19.876273 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:19.876706 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:19.876711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:19.876713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:19.877172 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:19.877353 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:19.877859 (+ 506us) server_negotiation.cc:299] Negotiation successful
0809 20:24:19.878149 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15087,"thread_start_us":15275,"threads_started":1}
I20260809 20:24:19.878643 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:19.844845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:47662)
0809 20:24:19.859957 (+ 15112us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:19.860035 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:19.860245 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:19.861011 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:19.861019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:19.861996 (+ 977us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:19.862459 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:19.862472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.864424 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.864432 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:19.866642 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:19.866661 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.867609 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.867617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:19.867802 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:19.868462 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:19.868484 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:19.871380 (+ 2896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:19.875492 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:19.875503 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:19.875528 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:19.875989 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:19.876393 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:19.876398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:19.876402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:19.876592 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:19.877334 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:19.877342 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:19.877559 (+ 217us) client_negotiation.cc:769] Sending connection context
0809 20:24:19.877755 (+ 196us) client_negotiation.cc:240] Negotiation successful
0809 20:24:19.878131 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14753,"thread_start_us":15569,"threads_started":1}
I20260809 20:24:19.881686 19657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:24:19 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307059, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.61@KRBTEST.COM for oryx/127.12.78.61@KRBTEST.COM
I20260809 20:24:19.896925 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:19.882610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:39915 (local address 127.0.0.1:40336)
0809 20:24:19.882780 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:19.882794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:19.882915 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:19.883205 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:19.883209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:19.883616 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:19.883817 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:19.883826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.884996 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.885001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:19.886295 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:19.886307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.887014 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.887019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:19.887258 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:19.888000 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:19.888017 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:19.890526 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:19.894068 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:19.894075 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:19.894079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:19.894537 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:19.894960 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:19.894966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:19.894970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:19.895146 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:19.896016 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:19.896019 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:19.896152 (+ 133us) client_negotiation.cc:769] Sending connection context
0809 20:24:19.896295 (+ 143us) client_negotiation.cc:240] Negotiation successful
0809 20:24:19.896602 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":38}
I20260809 20:24:19.897607 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:19.882480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.12.78.61:39915)
0809 20:24:19.882638 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:19.882642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:19.882857 (+ 215us) server_negotiation.cc:407] Connection header received
0809 20:24:19.883033 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:19.883044 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:19.883095 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:19.883192 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:19.883950 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.884809 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.886432 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.886864 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.887002 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:19.890701 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:19.890711 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:19.890714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:19.890742 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:19.893919 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:19.894660 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:19.894667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:19.894671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:19.894849 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:19.895312 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:19.895321 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:19.895327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:19.895885 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:19.896080 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:19.896452 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 20:24:19.896800 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":50}
I20260809 20:24:19.903299 19657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: Bootstrap starting.
I20260809 20:24:19.907171 19657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:19.908679 19657 log.cc:824] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:19.912251 19657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: No bootstrap required, opened a new log
I20260809 20:24:19.922009 19657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } }
I20260809 20:24:19.922457 19657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:19.922617 19657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dcfa89acf6426c930dddd0e37f1254, State: Initialized, Role: FOLLOWER
I20260809 20:24:19.923285 19657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } }
I20260809 20:24:19.924993 19664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } } }
I20260809 20:24:19.925558 19664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:24:19.926313 19657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 20:24:19.940109 19677 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 20:24:19.940326 19677 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 20:24:19.944272 19587 master_runner.cc:428] Detected that this master 45dcfa89acf6426c930dddd0e37f1254 is joining an existing cluster
I20260809 20:24:19.944440 19587 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.78.61:39915
I20260809 20:24:19.969441 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:19.950430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.12.78.61:39915)
0809 20:24:19.950653 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:19.950659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:19.952260 (+ 1601us) server_negotiation.cc:407] Connection header received
0809 20:24:19.952486 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:19.952492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:19.952579 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:19.952648 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:19.954365 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.955495 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.957956 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.958682 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.958783 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:19.961960 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:19.961967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:19.961972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:19.962006 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:19.966203 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:19.967069 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:19.967074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:19.967078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:19.967227 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:19.967678 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:19.967683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:19.967686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:19.968029 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:19.968172 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:19.968727 (+ 555us) server_negotiation.cc:299] Negotiation successful
0809 20:24:19.969047 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 20:24:19.970517 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:19.950874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:39915 (local address 127.0.0.1:40352)
0809 20:24:19.952167 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:19.952189 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:19.952344 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:19.952840 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:19.952844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:19.953812 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:19.954191 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:19.954202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.955714 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.955719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:19.957802 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:19.957817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.959098 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.959105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:19.959289 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:19.960214 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:19.960251 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:19.961741 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:19.966374 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:19.966379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:19.966382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:19.966930 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:19.967376 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:19.967381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:19.967384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:19.967566 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:19.968256 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:19.968261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:19.968404 (+ 143us) client_negotiation.cc:769] Sending connection context
0809 20:24:19.969711 (+ 1307us) client_negotiation.cc:240] Negotiation successful
0809 20:24:19.969993 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"mutex_wait_us":33,"thread_start_us":959,"threads_started":1}
I20260809 20:24:19.973937 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:19.949532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.12.78.62:39677)
0809 20:24:19.949726 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:19.949734 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:19.951312 (+ 1578us) server_negotiation.cc:407] Connection header received
0809 20:24:19.951706 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:19.951714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:19.951821 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:19.951930 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:19.954138 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.955492 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.958268 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.959127 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.959260 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:19.963113 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:19.963122 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:19.963129 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:19.963180 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:19.968221 (+ 5041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:19.969301 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:19.969309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:19.969315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:19.969506 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:19.970453 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:19.970461 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:19.970466 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:19.972589 (+ 2123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:19.972751 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:19.973337 (+ 586us) server_negotiation.cc:299] Negotiation successful
0809 20:24:19.973592 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 20:24:19.975029 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:19.949616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:47664)
0809 20:24:19.951157 (+ 1541us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:19.951212 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:19.951503 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:19.952167 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:19.952175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:19.953133 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:19.953878 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:19.953896 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.955732 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.955740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:19.958080 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:19.958099 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.959975 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.959989 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:19.960253 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:19.961036 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:19.961066 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:19.962520 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:19.968434 (+ 5914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:19.968441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:19.968448 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:19.969140 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:19.970039 (+ 899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:19.970046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:19.970051 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:19.970284 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:19.972831 (+ 2547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:19.972836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:19.973025 (+ 189us) client_negotiation.cc:769] Sending connection context
0809 20:24:19.974178 (+ 1153us) client_negotiation.cc:240] Negotiation successful
0809 20:24:19.974591 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"mutex_wait_us":1,"thread_start_us":1435,"threads_started":1}
Aug 09 20:24:19 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307057, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.62@KRBTEST.COM for oryx/127.12.78.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.12.78.60@KRBTEST.COM; defaulting to no policy
I20260809 20:24:20.014060 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:19.984535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:39915 (local address 127.0.0.1:40354)
0809 20:24:19.993161 (+ 8626us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:19.993222 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:19.993688 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:19.994598 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:19.994607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:19.995823 (+ 1216us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:19.996212 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:19.996227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.997969 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.997975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:19.999622 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:19.999640 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.000987 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.000994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.001239 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.002077 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:20.002103 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:20.005309 (+ 3206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:20.010036 (+ 4727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.010061 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.010070 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.010723 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.011278 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.011283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.011287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.011483 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.012375 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:20.012385 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:20.012818 (+ 433us) client_negotiation.cc:769] Sending connection context
0809 20:24:20.013050 (+ 232us) client_negotiation.cc:240] Negotiation successful
0809 20:24:20.013436 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8018,"thread_start_us":7837,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.78.60@KRBTEST.COM" created.
I20260809 20:24:20.014400 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:19.984760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.12.78.61:39915)
0809 20:24:19.985143 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:19.985149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:19.993582 (+ 8433us) server_negotiation.cc:407] Connection header received
0809 20:24:19.993990 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:19.993996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:19.994102 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:19.994258 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:19.996414 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:19.997680 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:19.999805 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.000577 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.000690 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.005591 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:20.005599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:20.005604 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:20.005656 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:20.009802 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.010891 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.010897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.010901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.011109 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.011634 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.011642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.011656 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.012170 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:20.012359 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:20.013201 (+ 842us) server_negotiation.cc:299] Negotiation successful
0809 20:24:20.013523 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260809 20:24:20.016587 19101 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.78.61:39915: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254"
member_type: NON_VOTER
last_known_addr {
host: "127.12.78.61"
port: 39915
}
attrs {
promote: true
}
}
dest_uuid: "ccd8c1717f7b4ce8b2489ede55f20522"
cas_config_opid_index: -1
I20260809 20:24:20.017774 19101 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true } }
I20260809 20:24:20.039896 19630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:20.040854 19690 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 45dcfa89acf6426c930dddd0e37f1254 (127.12.78.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true } } }
I20260809 20:24:20.044492 19630 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 FOLLOWER]: Refusing update from remote peer ccd8c1717f7b4ce8b2489ede55f20522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 20:24:20.044749 19690 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.78.61:39915
I20260809 20:24:20.051283 19690 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:20.051520 19689 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true } } }
I20260809 20:24:20.052556 19689 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: This master's current role is: LEADER
W20260809 20:24:20.053205 19609 tablet.cc:2369] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:24:20.061363 19587 master.cc:537] Master@127.12.78.61:39915 shutting down...
W20260809 20:24:20.064615 19085 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260809 20:24:20.074739 19085 consensus_peers.cc:487] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 -> Peer 45dcfa89acf6426c930dddd0e37f1254 (127.12.78.61:39915): Couldn't send request to peer 45dcfa89acf6426c930dddd0e37f1254. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.78.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.60.keytab.
Entry for principal oryx/127.12.78.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.60.keytab.
W20260809 20:24:20.077152 12600 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.12.78.60: real 0.103s user 0.003s sys 0.023s
I20260809 20:24:20.078781 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.60:38949
--webserver_interface=127.12.78.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.60.keytab
--principal=oryx/127.12.78.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:39677,127.12.78.61:39915,127.12.78.60:38949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:24:20.085517 19587 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:20.086140 19587 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:20.086410 19587 tablet_replica.cc:330] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: stopping tablet replica
I20260809 20:24:20.095510 19587 master.cc:552] Master@127.12.78.61:39915 shutdown complete.
I20260809 20:24:20.095777 19587 master_runner.cc:305] Clearing existing system tablet
I20260809 20:24:20.096339 19587 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:20.096442 19587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:20.105568 19587 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.000s sys 0.008s
I20260809 20:24:20.116926 19697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:20.120930 19587 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:20.121472 19587 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.000s sys 0.011s
I20260809 20:24:20.121619 19587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/wal
uuid: "45dcfa89acf6426c930dddd0e37f1254"
format_stamp: "Formatted at 2026-08-09 20:24:19 on dist-test-slave-rnrw"
server_key: "a298ba5291ecc2518838aefb66cb0847"
server_key_iv: "7572b90b71dc89b45bbe4b1d61a09285"
server_key_version: "encryptionkey@0"
I20260809 20:24:20.126952 19587 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:20.134518 19587 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 20:24:20.134835 19587 log.cc:1196] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 20:24:20.135648 19587 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: Deleting consensus metadata
I20260809 20:24:20.136010 19587 master_runner.cc:315] Copying system tablet from 127.12.78.62:39677
I20260809 20:24:20.160986 19587 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: tablet copy: Beginning tablet copy session from remote peer at address 127.12.78.62:39677
I20260809 20:24:20.179801 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.161946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.12.78.62:39677)
0809 20:24:20.162196 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:20.162203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:20.162786 (+ 583us) server_negotiation.cc:407] Connection header received
0809 20:24:20.163006 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:20.163012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:20.163109 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:20.163226 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:20.164654 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.165871 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.168217 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.168963 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.169107 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.172923 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:20.172930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:20.172934 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:20.172978 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:20.176706 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.177512 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.177517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.177521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.177685 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.178143 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.178147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.178150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.178519 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:20.178669 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:20.179204 (+ 535us) server_negotiation.cc:299] Negotiation successful
0809 20:24:20.179433 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 20:24:20.180013 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.161911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:47666)
0809 20:24:20.162657 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:20.162679 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:20.162848 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:20.163437 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:20.163443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:20.164032 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:20.164353 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.164368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.166104 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.166110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.168009 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.168039 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.169392 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.169400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.169585 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.170335 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:20.170361 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:20.172655 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:20.176840 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.176846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.176850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.177380 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.177818 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.177824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.177828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.178000 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.178661 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:20.178666 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:20.178917 (+ 251us) client_negotiation.cc:769] Sending connection context
0809 20:24:20.179077 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 20:24:20.179433 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":420,"threads_started":1}
I20260809 20:24:20.181170 19122 tablet_copy_service.cc:140] P ccd8c1717f7b4ce8b2489ede55f20522: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 45dcfa89acf6426c930dddd0e37f1254 ({username='oryx', principal='oryx/127.12.78.61@KRBTEST.COM'} at 127.0.0.1:47666)
I20260809 20:24:20.181486 19122 tablet_copy_service.cc:161] P ccd8c1717f7b4ce8b2489ede55f20522: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 45dcfa89acf6426c930dddd0e37f1254 at {username='oryx', principal='oryx/127.12.78.61@KRBTEST.COM'} at 127.0.0.1:47666: session id = 45dcfa89acf6426c930dddd0e37f1254-00000000000000000000000000000000
I20260809 20:24:20.185899 19122 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 20:24:20.188163 19587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 20:24:20.195151 19587 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: tablet copy: Starting download of 0 data blocks...
I20260809 20:24:20.195479 19587 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: tablet copy: Starting download of 1 WAL segments...
I20260809 20:24:20.198500 19587 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 20:24:20.203217 19122 tablet_copy_service.cc:342] P ccd8c1717f7b4ce8b2489ede55f20522: Request end of tablet copy session 45dcfa89acf6426c930dddd0e37f1254-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.78.61@KRBTEST.COM'} at 127.0.0.1:47666
I20260809 20:24:20.203455 19122 tablet_copy_service.cc:433] P ccd8c1717f7b4ce8b2489ede55f20522: ending tablet copy session 45dcfa89acf6426c930dddd0e37f1254-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 45dcfa89acf6426c930dddd0e37f1254
I20260809 20:24:20.216244 19587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:20.219158 19705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:20.220315 19706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:20.225427 19587 server_base.cc:1034] running on GCE node
W20260809 20:24:20.225620 19708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:20.226143 19587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:20.226882 19587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:20.228272 19587 hybrid_clock.cc:648] HybridClock initialized: now 1786307060228185 us; error 52 us; skew 500 ppm
I20260809 20:24:20.228521 19587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:20 dist-test-slave-rnrw krb5kdc[19054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307060, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:20.230809 19587 init.cc:377] Logged in from keytab as oryx/127.12.78.61@KRBTEST.COM (short username oryx)
I20260809 20:24:20.232172 19587 webserver.cc:460] Webserver started at http://127.12.78.61:33055/ using document root <none> and password file <none>
I20260809 20:24:20.232743 19587 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:20.232899 19587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:20.236343 19587 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260809 20:24:20.238399 19715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:20.238847 19587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 20:24:20.239022 19587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/wal
uuid: "45dcfa89acf6426c930dddd0e37f1254"
format_stamp: "Formatted at 2026-08-09 20:24:19 on dist-test-slave-rnrw"
server_key: "a298ba5291ecc2518838aefb66cb0847"
server_key_iv: "7572b90b71dc89b45bbe4b1d61a09285"
server_key_version: "encryptionkey@0"
I20260809 20:24:20.239236 19587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:20.249994 19587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:20.254125 19587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:20.291111 19587 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.61:39915
I20260809 20:24:20.291208 19767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.61:39915 every 8 connection(s)
I20260809 20:24:20.292366 19587 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-1/data/info.pb
I20260809 20:24:20.298324 19768 sys_catalog.cc:263] Verifying existing consensus state
I20260809 20:24:20.302573 19768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: Bootstrap starting.
I20260809 20:24:20.303280 19581 heartbeater.cc:503] Master 127.12.78.62:39677 was elected leader, sending a full tablet report...
I20260809 20:24:20.321238 19768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 20:24:20.321812 19768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: Bootstrap complete.
I20260809 20:24:20.322996 19768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true } }
I20260809 20:24:20.323302 19768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 45dcfa89acf6426c930dddd0e37f1254, State: Initialized, Role: LEARNER
I20260809 20:24:20.323578 19768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true } }
I20260809 20:24:20.324826 19768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:20.324617 19771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true } } }
I20260809 20:24:20.325295 19771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: This master's current role is: LEARNER
I20260809 20:24:20.336459 19783 catalog_manager.cc:1259] Loaded cluster ID: 958da1061cb0417788c3a84c366f3f25
I20260809 20:24:20.336694 19783 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: loading cluster ID for follower catalog manager: success
I20260809 20:24:20.340461 19783 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: acquiring CA information for follower catalog manager: success
I20260809 20:24:20.343364 19783 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 20:24:20.557552 19695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:20.557960 19695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:20.558197 19695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:20.567713 19695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:20.567881 19695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:20.567991 19695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:20.568127 19695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:20.568213 19695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:24:20.580768 19695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:39677,127.12.78.61:39915,127.12.78.60:38949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.60.keytab
--principal=oryx/127.12.78.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.60:38949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.78.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:20.582497 19695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:20.584019 19695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:20.598364 19788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:20.598665 19789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:20.598963 19791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:20.599879 19695 server_base.cc:1034] running on GCE node
I20260809 20:24:20.600787 19695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:20.602403 19695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:20.603703 19695 hybrid_clock.cc:648] HybridClock initialized: now 1786307060603592 us; error 52 us; skew 500 ppm
I20260809 20:24:20.604154 19695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:20 dist-test-slave-rnrw krb5kdc[19054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307060, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:20.607641 19695 init.cc:377] Logged in from keytab as oryx/127.12.78.60@KRBTEST.COM (short username oryx)
I20260809 20:24:20.609858 19695 webserver.cc:460] Webserver started at http://127.12.78.60:38755/ using document root <none> and password file <none>
I20260809 20:24:20.610759 19695 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:20.610888 19695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:20.611214 19695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:20.614653 19695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/data/instance:
uuid: "20eae6ac52e34db5abb5c061cf4051ca"
format_stamp: "Formatted at 2026-08-09 20:24:20 on dist-test-slave-rnrw"
server_key: "a6f08ef426234946f025076674463a8c"
server_key_iv: "12f6f202726f18a73b7ac0edad9411bd"
server_key_version: "encryptionkey@0"
I20260809 20:24:20.615511 19695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/wal/instance:
uuid: "20eae6ac52e34db5abb5c061cf4051ca"
format_stamp: "Formatted at 2026-08-09 20:24:20 on dist-test-slave-rnrw"
server_key: "a6f08ef426234946f025076674463a8c"
server_key_iv: "12f6f202726f18a73b7ac0edad9411bd"
server_key_version: "encryptionkey@0"
I20260809 20:24:20.621760 19695 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260809 20:24:20.626789 19803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:20.628404 19695 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 20:24:20.628710 19695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/wal
uuid: "20eae6ac52e34db5abb5c061cf4051ca"
format_stamp: "Formatted at 2026-08-09 20:24:20 on dist-test-slave-rnrw"
server_key: "a6f08ef426234946f025076674463a8c"
server_key_iv: "12f6f202726f18a73b7ac0edad9411bd"
server_key_version: "encryptionkey@0"
I20260809 20:24:20.628989 19695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:20.630757 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.623075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:39915 (local address 127.0.0.1:40356)
0809 20:24:20.623606 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:20.623628 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:20.623783 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:20.624497 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:20.624502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:20.624574 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:20.624873 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.624889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.626787 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.626792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.628361 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.628375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.629854 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.629859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.630037 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.630117 (+ 80us) client_negotiation.cc:769] Sending connection context
0809 20:24:20.630232 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 20:24:20.630384 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":258,"threads_started":1}
I20260809 20:24:20.631060 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.623382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40356 (local address 127.12.78.61:39915)
0809 20:24:20.624140 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:20.624145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:20.624162 (+ 17us) server_negotiation.cc:407] Connection header received
0809 20:24:20.624223 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:20.624227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:20.624294 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:20.624394 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:20.625096 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.626593 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.628663 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.629670 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.629828 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.630038 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:20.630260 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 20:24:20.630393 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":518,"threads_started":1}
I20260809 20:24:20.632306 19742 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260809 20:24:20.632921 19771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccd8c1717f7b4ce8b2489ede55f20522. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true } } }
I20260809 20:24:20.633285 19771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: This master's current role is: LEARNER
I20260809 20:24:20.660116 19695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:20.665513 19695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:20.665833 19695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:20.697747 19695 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.60:38949
I20260809 20:24:20.697829 19855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.60:38949 every 8 connection(s)
I20260809 20:24:20.699468 19695 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/data/info.pb
I20260809 20:24:20.703634 19856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:20.704574 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 19695
I20260809 20:24:20.714730 19856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:24:20 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.78.60@KRBTEST.COM
Aug 09 20:24:20 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307060, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.60@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
I20260809 20:24:20.735620 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.707213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51846 (local address 127.12.78.60:38949)
0809 20:24:20.708702 (+ 1489us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:20.708713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:20.708776 (+ 63us) server_negotiation.cc:407] Connection header received
0809 20:24:20.711171 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:20.711193 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:20.711577 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:20.711990 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:20.714604 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.716390 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.718421 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.719889 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.720082 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.725156 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:20.725164 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:20.725175 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:20.725214 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:20.729348 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.730207 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.730213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.730220 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.730448 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.731096 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.731099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.731101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.731695 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:20.731989 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:20.733387 (+ 1398us) server_negotiation.cc:299] Negotiation successful
0809 20:24:20.733810 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1154,"thread_start_us":828,"threads_started":1}
I20260809 20:24:20.739055 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.717906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.12.78.62:39677)
0809 20:24:20.718744 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:20.718751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:20.719474 (+ 723us) server_negotiation.cc:407] Connection header received
0809 20:24:20.719723 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:20.719728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:20.719798 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:20.720025 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:20.721845 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.722704 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.724685 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.725175 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.725331 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.730145 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:20.730153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:20.730158 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:20.730206 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:20.735060 (+ 4854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.736406 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.736412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.736416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.736705 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.737201 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.737206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.737210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.737551 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:20.737723 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:20.738338 (+ 615us) server_negotiation.cc:299] Negotiation successful
0809 20:24:20.738571 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":501,"threads_started":1}
I20260809 20:24:20.739144 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.717956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:47674)
0809 20:24:20.719237 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:20.719292 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:20.719566 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:20.720142 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:20.720150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:20.721139 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:20.721683 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.721694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.722908 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.722914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.724459 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.724476 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.725396 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.725410 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.725683 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.726509 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:20.726534 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:20.729928 (+ 3394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:20.735261 (+ 5333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.735270 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.735277 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.736262 (+ 985us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.736834 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.736840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.736845 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.737065 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.737699 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:20.737707 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:20.737958 (+ 251us) client_negotiation.cc:769] Sending connection context
0809 20:24:20.738212 (+ 254us) client_negotiation.cc:240] Negotiation successful
0809 20:24:20.738576 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"thread_start_us":719,"threads_started":1}
I20260809 20:24:20.740957 19856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:24:20 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307060, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.60@KRBTEST.COM for oryx/127.12.78.61@KRBTEST.COM
I20260809 20:24:20.757334 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.741662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:39915 (local address 127.0.0.1:40370)
0809 20:24:20.741910 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:20.741962 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:20.742070 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:20.742486 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:20.742489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:20.742901 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:20.743123 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.743131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.744299 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.744303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.745795 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.745809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.746678 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.746683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.746902 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.747459 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:20.747476 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:20.750541 (+ 3065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:20.754439 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.754445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.754450 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.754973 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.755424 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.755430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.755433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.755620 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.756188 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:20.756193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:20.756366 (+ 173us) client_negotiation.cc:769] Sending connection context
0809 20:24:20.756512 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 20:24:20.756914 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 20:24:20.757375 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.742011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.12.78.61:39915)
0809 20:24:20.742188 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:20.742192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:20.742208 (+ 16us) server_negotiation.cc:407] Connection header received
0809 20:24:20.742261 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:20.742264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:20.742327 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:20.742408 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:20.743254 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.744119 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.745953 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.746528 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.746647 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.750732 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:20.750737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:20.750740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:20.750773 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:20.754297 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.755107 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.755114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.755122 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.755306 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.755734 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.755741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.755744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.756063 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:20.756222 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:20.756642 (+ 420us) server_negotiation.cc:299] Negotiation successful
0809 20:24:20.756914 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 20:24:20.758574 19856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:24:20 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307060, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.60@KRBTEST.COM for oryx/127.12.78.60@KRBTEST.COM
I20260809 20:24:20.773891 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.759572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.12.78.60:38949)
0809 20:24:20.759722 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:20.759726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:20.759741 (+ 15us) server_negotiation.cc:407] Connection header received
0809 20:24:20.759814 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:20.759817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:20.759869 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:20.759943 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:20.760956 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.761750 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.763404 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.763886 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.763984 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.767330 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:20.767335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:20.767338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:20.767364 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:20.770415 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.771167 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.771174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.771177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.771329 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.771809 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.771817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.771820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.772197 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:20.772352 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:20.772844 (+ 492us) server_negotiation.cc:299] Negotiation successful
0809 20:24:20.773266 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":54}
I20260809 20:24:20.773924 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.759350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38949 (local address 127.0.0.1:51852)
0809 20:24:20.759625 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:20.759642 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:20.759729 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:20.759989 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:20.759993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:20.760510 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:20.760830 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.760840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.761871 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.761875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.763240 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.763253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.764050 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.764055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.764228 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.764805 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:20.764823 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:20.767154 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:20.770576 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.770581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.770585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.771030 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.771438 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.771445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.771449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.771696 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.772313 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:20.772318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:20.772613 (+ 295us) client_negotiation.cc:769] Sending connection context
0809 20:24:20.772770 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 20:24:20.773052 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 20:24:20.782289 19856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: Bootstrap starting.
I20260809 20:24:20.787647 19856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:20.789176 19856 log.cc:824] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:20.792734 19856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: No bootstrap required, opened a new log
I20260809 20:24:20.801550 19856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } }
I20260809 20:24:20.801880 19856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:20.802028 19856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20eae6ac52e34db5abb5c061cf4051ca, State: Initialized, Role: FOLLOWER
I20260809 20:24:20.802652 19856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } }
I20260809 20:24:20.803861 19863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } } }
I20260809 20:24:20.804368 19863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:24:20.804893 19856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: configured and running, proceeding with master startup.
W20260809 20:24:20.818179 19874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 20:24:20.818424 19874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 20:24:20.822829 19695 master_runner.cc:428] Detected that this master 20eae6ac52e34db5abb5c061cf4051ca is joining an existing cluster
I20260809 20:24:20.823055 19695 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.78.60:38949
I20260809 20:24:20.842878 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 19164
I20260809 20:24:20.866282 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.829231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.12.78.62:39677)
0809 20:24:20.829425 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:20.829429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:20.834110 (+ 4681us) server_negotiation.cc:407] Connection header received
0809 20:24:20.834510 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:20.834516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:20.834607 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:20.834696 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:20.837123 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.838561 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.852078 (+ 13517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.852980 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.853129 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.857397 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:20.857408 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:20.857415 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:20.857464 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:20.862464 (+ 5000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.863320 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.863327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.863330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.863548 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.864096 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.864101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.864104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.864582 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:20.864762 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:20.865469 (+ 707us) server_negotiation.cc:299] Negotiation successful
0809 20:24:20.865846 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 20:24:20.867982 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.828604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:47686)
0809 20:24:20.833469 (+ 4865us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:20.833490 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:20.834320 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:20.835010 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:20.835015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:20.836353 (+ 1338us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:20.836925 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.836940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.838864 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.838869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.842635 (+ 3766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.842659 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.853414 (+ 10755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.853420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.853669 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.854563 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:20.854587 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:20.857091 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:20.862654 (+ 5563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.862658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.862663 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.863168 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.863716 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.863721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.863724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.863942 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.864854 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:20.864857 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:20.865137 (+ 280us) client_negotiation.cc:769] Sending connection context
0809 20:24:20.866634 (+ 1497us) client_negotiation.cc:240] Negotiation successful
0809 20:24:20.867063 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4670,"thread_start_us":4476,"threads_started":1}
I20260809 20:24:20.869190 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.828921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.12.78.61:39915)
0809 20:24:20.829137 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:20.829143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:20.840600 (+ 11457us) server_negotiation.cc:407] Connection header received
0809 20:24:20.840963 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:20.840969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:20.841065 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:20.841141 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:20.843498 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.844950 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.848658 (+ 3708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.849285 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.849386 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.856681 (+ 7295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:20.856691 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:20.856697 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:20.856742 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:20.861222 (+ 4480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.862607 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.862614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.862618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.862860 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.863878 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.863883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.863887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.864310 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:20.864466 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:20.867816 (+ 3350us) server_negotiation.cc:299] Negotiation successful
0809 20:24:20.868647 (+ 831us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 20:24:20.871209 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.828761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:39915 (local address 127.0.0.1:40376)
0809 20:24:20.840357 (+ 11596us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:20.840379 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:20.840714 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:20.841388 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:20.841393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:20.842419 (+ 1026us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:20.843295 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.843310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.845404 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.845410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.847460 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.847474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.849708 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.849716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.849901 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.853704 (+ 3803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:20.853819 (+ 115us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:20.856364 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:20.861418 (+ 5054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.861424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.861429 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.862032 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.863039 (+ 1007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.863044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.863048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.863222 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.864581 (+ 1359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:20.864587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:20.864848 (+ 261us) client_negotiation.cc:769] Sending connection context
0809 20:24:20.869684 (+ 4836us) client_negotiation.cc:240] Negotiation successful
0809 20:24:20.870265 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11420,"spinlock_wait_cycles":28544,"thread_start_us":19083,"threads_started":1}
Aug 09 20:24:20 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.78.1@KRBTEST.COM
I20260809 20:24:20.884122 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.844860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.12.78.1:45283)
0809 20:24:20.857345 (+ 12485us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:20.857356 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:20.857392 (+ 36us) server_negotiation.cc:407] Connection header received
0809 20:24:20.857472 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:20.857483 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:20.857827 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:20.861001 (+ 3174us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:20.861603 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.863479 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.866187 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.867537 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.872299 (+ 4762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.875802 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:20.875819 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:20.875838 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:20.875880 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:20.879906 (+ 4026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.881163 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.881175 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.881178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.881416 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.881948 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.881953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.881956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.882424 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:20.882883 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:20.883236 (+ 353us) server_negotiation.cc:299] Negotiation successful
0809 20:24:20.883609 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12245,"thread_start_us":13216,"threads_started":1}
I20260809 20:24:20.885684 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.834006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.12.78.60:38949)
0809 20:24:20.834233 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:20.834240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:20.848430 (+ 14190us) server_negotiation.cc:407] Connection header received
0809 20:24:20.848629 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:20.848637 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:20.848734 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:20.848825 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:20.851960 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.853412 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.865127 (+ 11715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.866096 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.866203 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.877338 (+ 11135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:20.877346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:20.877350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:20.877399 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:20.881992 (+ 4593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.882930 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.882935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.882938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.883087 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.883551 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.883556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.883559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.884133 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:20.884268 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:20.885080 (+ 812us) server_negotiation.cc:299] Negotiation successful
0809 20:24:20.885310 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 20:24:20.885816 19297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:20.886757 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.833925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38949 (local address 127.0.0.1:51858)
0809 20:24:20.848181 (+ 14256us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:20.848200 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:20.848466 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:20.850288 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:20.850293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:20.851027 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:20.851780 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.851793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.854915 (+ 3122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.854924 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.863701 (+ 8777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.863713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.866900 (+ 3187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.866907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.867165 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.871755 (+ 4590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:20.871800 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:20.877155 (+ 5355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:20.882163 (+ 5008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.882169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.882173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.882766 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.883227 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.883231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.883245 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.883411 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.884348 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:20.884352 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:20.884533 (+ 181us) client_negotiation.cc:769] Sending connection context
0809 20:24:20.884736 (+ 203us) client_negotiation.cc:240] Negotiation successful
0809 20:24:20.886184 (+ 1448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14117,"thread_start_us":5988,"threads_started":1}
Aug 09 20:24:20 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307057, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.62@KRBTEST.COM for oryx/127.12.78.60@KRBTEST.COM
I20260809 20:24:20.915778 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.875935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38949 (local address 127.0.0.1:51862)
0809 20:24:20.876294 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:20.876319 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:20.876459 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:20.877428 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:20.877436 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:20.878165 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:20.878381 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.878396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.898968 (+ 20572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.898975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.900752 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:20.900774 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.902325 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.902332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:20.902582 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.903364 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:20.903391 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:20.907189 (+ 3798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:20.912128 (+ 4939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.912134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.912139 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.912805 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.913376 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:20.913382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:20.913386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:20.913607 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:20.914508 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:20.914514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:20.914765 (+ 251us) client_negotiation.cc:769] Sending connection context
0809 20:24:20.914909 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 20:24:20.915251 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260809 20:24:20.916118 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.875978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.12.78.60:38949)
0809 20:24:20.876841 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:20.876848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:20.876870 (+ 22us) server_negotiation.cc:407] Connection header received
0809 20:24:20.876958 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:20.876962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:20.877082 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:20.878550 (+ 1468us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:20.897224 (+ 18674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.898717 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.900970 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:20.902117 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:20.902279 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:20.907517 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:20.907529 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:20.907535 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:20.907579 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:20.911946 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.912947 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.912953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.912964 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.913222 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:20.913750 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:20.913755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:20.913759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:20.914328 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:20.914523 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:20.915116 (+ 593us) server_negotiation.cc:299] Negotiation successful
0809 20:24:20.915378 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":518,"threads_started":1}
I20260809 20:24:20.917944 19101 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.78.60:38949: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca"
member_type: NON_VOTER
last_known_addr {
host: "127.12.78.60"
port: 38949
}
attrs {
promote: true
}
}
dest_uuid: "ccd8c1717f7b4ce8b2489ede55f20522"
cas_config_opid_index: 5
W20260809 20:24:20.918310 19809 tablet.cc:2369] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:24:20.919126 19101 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } }
I20260809 20:24:20.921006 19798 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 20eae6ac52e34db5abb5c061cf4051ca (127.12.78.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } } }
I20260809 20:24:20.921875 19742 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 LEARNER]: Refusing update from remote peer ccd8c1717f7b4ce8b2489ede55f20522: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 20:24:20.926065 19831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:20.932273 19831 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 1 FOLLOWER]: Refusing update from remote peer ccd8c1717f7b4ce8b2489ede55f20522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 20:24:20.932768 19887 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 20:24:20.940681 19798 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.78.60:38949
I20260809 20:24:20.940914 19888 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 20:24:20.941582 19887 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } } }
I20260809 20:24:20.942019 19887 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:20.943822 19742 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 20eae6ac52e34db5abb5c061cf4051ca (127.12.78.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } } }
I20260809 20:24:20.955129 19695 master.cc:537] Master@127.12.78.60:38949 shutting down...
I20260809 20:24:20.956272 19771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: NON_VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: true } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } } }
I20260809 20:24:20.956617 19771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: This master's current role is: LEARNER
I20260809 20:24:20.957823 19798 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522: attempting to promote NON_VOTER 45dcfa89acf6426c930dddd0e37f1254 to VOTER
I20260809 20:24:20.958894 19798 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } }
I20260809 20:24:20.963997 19742 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 LEARNER]: Refusing update from remote peer ccd8c1717f7b4ce8b2489ede55f20522: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 20:24:20.964874 19889 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 20:24:20.971072 19798 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 45dcfa89acf6426c930dddd0e37f1254 (127.12.78.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } } }
I20260809 20:24:20.971371 19741 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 45dcfa89acf6426c930dddd0e37f1254 (127.12.78.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } } }
I20260809 20:24:20.972283 19695 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:20.972704 19695 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:20.972899 19695 tablet_replica.cc:330] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: stopping tablet replica
I20260809 20:24:20.975821 19771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } } }
I20260809 20:24:20.976085 19771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:24:20.976365 19892 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } } }
I20260809 20:24:20.977022 19892 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:20.975936 19798 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } } }
I20260809 20:24:20.977458 19798 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:20.979745 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:20.979158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38949 (local address 127.0.0.1:51876)
0809 20:24:20.979387 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:20.979526 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 20:24:20.980002 19087 consensus_peers.cc:487] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 -> Peer 20eae6ac52e34db5abb5c061cf4051ca (127.12.78.60:38949): Couldn't send request to peer 20eae6ac52e34db5abb5c061cf4051ca. Status: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 20:24:20.982548 19695 master.cc:552] Master@127.12.78.60:38949 shutdown complete.
I20260809 20:24:20.982642 19695 master_runner.cc:305] Clearing existing system tablet
I20260809 20:24:20.983069 19695 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:20.983172 19695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:20.986342 19695 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260809 20:24:20.988046 19895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:20.988467 19695 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:20.988763 19695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260809 20:24:20.988867 19695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/wal
uuid: "20eae6ac52e34db5abb5c061cf4051ca"
format_stamp: "Formatted at 2026-08-09 20:24:20 on dist-test-slave-rnrw"
server_key: "a6f08ef426234946f025076674463a8c"
server_key_iv: "12f6f202726f18a73b7ac0edad9411bd"
server_key_version: "encryptionkey@0"
I20260809 20:24:20.994222 19695 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:21.001560 19695 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 20:24:21.001755 19695 log.cc:1196] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 20:24:21.002252 19695 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: Deleting consensus metadata
I20260809 20:24:21.002529 19695 master_runner.cc:315] Copying system tablet from 127.12.78.62:39677
I20260809 20:24:21.005753 19695 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: tablet copy: Beginning tablet copy session from remote peer at address 127.12.78.62:39677
I20260809 20:24:21.024883 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:21.006715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.12.78.62:39677)
0809 20:24:21.006947 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:21.006954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:21.009259 (+ 2305us) server_negotiation.cc:407] Connection header received
0809 20:24:21.009493 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:21.009499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:21.009612 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:21.009735 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:21.010825 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.011933 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.013815 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.014497 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.014620 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:21.017101 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:21.017108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:21.017113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:21.017153 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:21.021389 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:21.022414 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:21.022420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:21.022424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:21.022585 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:21.023123 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:21.023128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:21.023131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:21.023462 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:21.023595 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:21.024218 (+ 623us) server_negotiation.cc:299] Negotiation successful
0809 20:24:21.024452 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260809 20:24:21.025203 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:21.006624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:47688)
0809 20:24:21.008941 (+ 2317us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:21.008962 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:21.009328 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:21.009780 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:21.009785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:21.010371 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:21.010645 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:21.010658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.012122 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.012129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:21.013657 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:21.013673 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.014664 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.014670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:21.014860 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:21.015612 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:21.015636 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:21.016886 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:21.021546 (+ 4660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:21.021552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:21.021556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:21.022246 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:21.022704 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:21.022710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:21.022713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:21.022966 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:21.023598 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:21.023603 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:21.023911 (+ 308us) client_negotiation.cc:769] Sending connection context
0809 20:24:21.024074 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 20:24:21.024515 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2157,"thread_start_us":2054,"threads_started":1}
I20260809 20:24:21.026134 19122 tablet_copy_service.cc:140] P ccd8c1717f7b4ce8b2489ede55f20522: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 20eae6ac52e34db5abb5c061cf4051ca ({username='oryx', principal='oryx/127.12.78.60@KRBTEST.COM'} at 127.0.0.1:47688)
I20260809 20:24:21.026408 19122 tablet_copy_service.cc:161] P ccd8c1717f7b4ce8b2489ede55f20522: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 20eae6ac52e34db5abb5c061cf4051ca at {username='oryx', principal='oryx/127.12.78.60@KRBTEST.COM'} at 127.0.0.1:47688: session id = 20eae6ac52e34db5abb5c061cf4051ca-00000000000000000000000000000000
I20260809 20:24:21.030754 19122 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 20:24:21.032838 19695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 20:24:21.034380 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 19164
I20260809 20:24:21.041603 19695 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: tablet copy: Starting download of 0 data blocks...
I20260809 20:24:21.041931 19695 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: tablet copy: Starting download of 1 WAL segments...
I20260809 20:24:21.044168 19695 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 20:24:21.047346 19122 tablet_copy_service.cc:342] P ccd8c1717f7b4ce8b2489ede55f20522: Request end of tablet copy session 20eae6ac52e34db5abb5c061cf4051ca-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.78.60@KRBTEST.COM'} at 127.0.0.1:47688
I20260809 20:24:21.047497 19122 tablet_copy_service.cc:433] P ccd8c1717f7b4ce8b2489ede55f20522: ending tablet copy session 20eae6ac52e34db5abb5c061cf4051ca-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 20eae6ac52e34db5abb5c061cf4051ca
I20260809 20:24:21.057403 19695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:21.060894 19902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:21.061147 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:45283
--local_ip_for_outbound_sockets=127.12.78.1
--tserver_master_addrs=127.12.78.62:39677
--webserver_port=35835
--webserver_interface=127.12.78.1
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab
--principal=oryx/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.78.62:39677,127.12.78.61:39915
--tserver_master_addrs=127.12.78.62:39677,127.12.78.61:39915,127.12.78.60:38949 with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:21.066329 19903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:21.070760 19905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:21.073196 19695 server_base.cc:1034] running on GCE node
I20260809 20:24:21.073524 19695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:21.074245 19695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:21.075488 19695 hybrid_clock.cc:648] HybridClock initialized: now 1786307061075408 us; error 39 us; skew 500 ppm
I20260809 20:24:21.075789 19695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:21 dist-test-slave-rnrw krb5kdc[19054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307061, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:21.077934 19695 init.cc:377] Logged in from keytab as oryx/127.12.78.60@KRBTEST.COM (short username oryx)
I20260809 20:24:21.083683 19695 webserver.cc:460] Webserver started at http://127.12.78.60:37313/ using document root <none> and password file <none>
I20260809 20:24:21.084003 19695 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:21.084117 19695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:21.086633 19695 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 20:24:21.088303 19916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:21.088732 19695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260809 20:24:21.088873 19695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/wal
uuid: "20eae6ac52e34db5abb5c061cf4051ca"
format_stamp: "Formatted at 2026-08-09 20:24:20 on dist-test-slave-rnrw"
server_key: "a6f08ef426234946f025076674463a8c"
server_key_iv: "12f6f202726f18a73b7ac0edad9411bd"
server_key_version: "encryptionkey@0"
I20260809 20:24:21.089077 19695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:21.099480 19695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:21.103030 19695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:21.141686 19695 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.60:38949
I20260809 20:24:21.141723 19969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.60:38949 every 8 connection(s)
I20260809 20:24:21.142616 19695 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/master-2/data/info.pb
I20260809 20:24:21.149952 19970 sys_catalog.cc:263] Verifying existing consensus state
I20260809 20:24:21.155714 19970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: Bootstrap starting.
I20260809 20:24:21.171020 19970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 20:24:21.171550 19970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: Bootstrap complete.
I20260809 20:24:21.172472 19970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } }
I20260809 20:24:21.172794 19970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20eae6ac52e34db5abb5c061cf4051ca, State: Initialized, Role: LEARNER
I20260809 20:24:21.173048 19970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } }
I20260809 20:24:21.175390 19970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:21.175031 19973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } } }
I20260809 20:24:21.175637 19973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: This master's current role is: LEARNER
I20260809 20:24:21.194307 19984 catalog_manager.cc:1259] Loaded cluster ID: 958da1061cb0417788c3a84c366f3f25
I20260809 20:24:21.194437 19984 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: loading cluster ID for follower catalog manager: success
I20260809 20:24:21.197335 19984 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: acquiring CA information for follower catalog manager: success
I20260809 20:24:21.199090 19984 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 20:24:21.547040 19907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:21.547329 19907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:21.547538 19907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:21.555984 19907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:21.556180 19907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:21.556444 19907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:24:21.568133 19907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab
--principal=oryx/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:45283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=35835
--tserver_master_addrs=127.12.78.62:39677,127.12.78.61:39915,127.12.78.60:38949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:21.569693 19907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:21.571074 19907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 20:24:21.583669 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:21.575041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38949 (local address 127.0.0.1:51884)
0809 20:24:21.575577 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:21.575600 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:21.575764 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:21.576342 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:21.576346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:21.576376 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:21.576646 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:21.576657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.578596 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.578600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:21.580742 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:21.580760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.582994 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.582999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:21.583204 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:21.583230 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 20:24:21.583321 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 20:24:21.583430 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":312,"threads_started":1}
I20260809 20:24:21.584005 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:21.575265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.12.78.60:38949)
0809 20:24:21.576023 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:21.576027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:21.576043 (+ 16us) server_negotiation.cc:407] Connection header received
0809 20:24:21.576105 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:21.576109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:21.576182 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:21.576314 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:21.576837 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.578396 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.581068 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.582765 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.582899 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:21.583100 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:21.583375 (+ 275us) server_negotiation.cc:299] Negotiation successful
0809 20:24:21.583583 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":470,"threads_started":1}
I20260809 20:24:21.585170 19944 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260809 20:24:21.585934 19973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccd8c1717f7b4ce8b2489ede55f20522. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } } }
I20260809 20:24:21.586387 19973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: This master's current role is: LEARNER
W20260809 20:24:21.589581 19991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:21.589893 19992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:21.589936 19995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:21.594408 19907 server_base.cc:1034] running on GCE node
I20260809 20:24:21.595247 19907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:21.596477 19907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:21.597782 19907 hybrid_clock.cc:648] HybridClock initialized: now 1786307061597690 us; error 59 us; skew 500 ppm
I20260809 20:24:21.598158 19907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:21 dist-test-slave-rnrw krb5kdc[19054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307061, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:21.602679 19907 init.cc:377] Logged in from keytab as oryx/127.12.78.1@KRBTEST.COM (short username oryx)
I20260809 20:24:21.604712 19907 webserver.cc:460] Webserver started at http://127.12.78.1:35835/ using document root <none> and password file <none>
I20260809 20:24:21.605752 19907 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:21.605993 19907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:21.612012 19907 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 20:24:21.615334 20002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:21.616901 19907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 20:24:21.617080 19907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "24068fc4286f45a4bfe48c531efac812"
format_stamp: "Formatted at 2026-08-09 20:24:18 on dist-test-slave-rnrw"
server_key: "c6389f4303735fb0a9de1ca6ab70fbe4"
server_key_iv: "9648b249794fc11981a1804be59ec1c1"
server_key_version: "encryptionkey@0"
I20260809 20:24:21.618305 19907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:21.638417 19907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:21.643781 19907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:21.644146 19907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:21.645673 19907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:21.647219 19907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:21.647354 19907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:21.647550 19907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:21.647684 19907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:21.706754 19907 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:45283
I20260809 20:24:21.707070 20119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:45283 every 8 connection(s)
I20260809 20:24:21.708348 19907 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:24:21.715564 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 19907
I20260809 20:24:21.715950 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 19309
Aug 09 20:24:21 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.78.2@KRBTEST.COM
Aug 09 20:24:21 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307061, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.61@KRBTEST.COM
Aug 09 20:24:21 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307061, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
Aug 09 20:24:21 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307061, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.60@KRBTEST.COM
I20260809 20:24:21.747429 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:21.712103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:42435 (local address 127.12.78.61:39915)
0809 20:24:21.712805 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:21.712811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:21.714221 (+ 1410us) server_negotiation.cc:407] Connection header received
0809 20:24:21.720308 (+ 6087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:21.720314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:21.720420 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:21.720534 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:21.724191 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.725582 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.729444 (+ 3862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.730508 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.730739 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:21.738167 (+ 7428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:21.738175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:21.738180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:21.738259 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:21.742859 (+ 4600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:21.744214 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:21.744219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:21.744223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:21.744416 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:21.745136 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:21.745142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:21.745147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:21.745539 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:21.745677 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:21.746778 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0809 20:24:21.747032 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":357,"threads_started":1}
I20260809 20:24:21.748870 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:21.712058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:60059 (local address 127.12.78.60:38949)
0809 20:24:21.712276 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:21.712284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:21.715430 (+ 3146us) server_negotiation.cc:407] Connection header received
0809 20:24:21.719803 (+ 4373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:21.719808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:21.719889 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:21.720000 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:21.724171 (+ 4171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.725657 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.729688 (+ 4031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.730731 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.730840 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:21.740575 (+ 9735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:21.740585 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:21.740591 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:21.740644 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:21.745359 (+ 4715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:21.746466 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:21.746473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:21.746476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:21.746641 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:21.747103 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:21.747108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:21.747111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:21.747455 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:21.747622 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:21.748296 (+ 674us) server_negotiation.cc:299] Negotiation successful
0809 20:24:21.748534 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260809 20:24:21.749316 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:21.715710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:59485 (local address 127.12.78.62:39677)
0809 20:24:21.717877 (+ 2167us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:21.717894 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:21.717953 (+ 59us) server_negotiation.cc:407] Connection header received
0809 20:24:21.719802 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:21.719808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:21.719889 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:21.719991 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:21.724178 (+ 4187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.725668 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.729698 (+ 4030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.730749 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.730839 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:21.740478 (+ 9639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:21.740485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:21.740491 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:21.740532 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:21.745439 (+ 4907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:21.746471 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:21.746477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:21.746481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:21.746621 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:21.747144 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:21.747149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:21.747152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:21.747583 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:21.747713 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:21.748476 (+ 763us) server_negotiation.cc:299] Negotiation successful
0809 20:24:21.748764 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":470,"threads_started":1}
I20260809 20:24:21.749955 20131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:21.717458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.12.78.2:37183)
0809 20:24:21.718799 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:21.718805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:21.718827 (+ 22us) server_negotiation.cc:407] Connection header received
0809 20:24:21.718884 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:21.718890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:21.719101 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:21.719374 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:21.722472 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.724825 (+ 2353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.727468 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.728627 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.738072 (+ 9445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:21.738454 (+ 382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:21.738464 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:21.738476 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:21.738501 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:21.744900 (+ 6399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:21.746057 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:21.746066 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:21.746071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:21.746245 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:21.746769 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:21.746774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:21.746778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:21.747305 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:21.748965 (+ 1660us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:21.749131 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 20:24:21.749487 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1166,"thread_start_us":929,"threads_started":1}
I20260809 20:24:21.750540 20127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:21.712773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38949 (local address 127.12.78.1:60059)
0809 20:24:21.715198 (+ 2425us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:21.715219 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:21.719380 (+ 4161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:21.720338 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:21.720349 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:21.722408 (+ 2059us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:21.723989 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:21.723999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.725913 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.725920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:21.729500 (+ 3580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:21.729521 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.731421 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.731428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:21.731964 (+ 536us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:21.737353 (+ 5389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:21.737369 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:21.740305 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:21.745518 (+ 5213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:21.745523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:21.745526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:21.746307 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:21.746775 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:21.746780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:21.746784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:21.746955 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:21.747685 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:21.747689 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:21.747832 (+ 143us) client_negotiation.cc:769] Sending connection context
0809 20:24:21.747970 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 20:24:21.749362 (+ 1392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2239,"mutex_wait_us":1661,"spinlock_wait_cycles":37632,"thread_start_us":1946,"threads_started":1}
I20260809 20:24:21.750692 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:21.715334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.1:59485)
0809 20:24:21.717578 (+ 2244us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:21.717609 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:21.719639 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:21.720345 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:21.720349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:21.722253 (+ 1904us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:21.723924 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:21.723974 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.725876 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.725882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:21.729546 (+ 3664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:21.729558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.731372 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.731393 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:21.731962 (+ 569us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:21.733239 (+ 1277us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:21.733282 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:21.740197 (+ 6915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:21.745621 (+ 5424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:21.745628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:21.745632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:21.746332 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:21.746769 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:21.746774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:21.746777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:21.747007 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:21.747799 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:21.747803 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:21.748109 (+ 306us) client_negotiation.cc:769] Sending connection context
0809 20:24:21.749593 (+ 1484us) client_negotiation.cc:240] Negotiation successful
0809 20:24:21.749951 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2079,"mutex_wait_us":1030,"thread_start_us":1516,"threads_started":1}
I20260809 20:24:21.751742 19437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:21.755960 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:21.712742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:39915 (local address 127.12.78.1:42435)
0809 20:24:21.714027 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:21.714081 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:21.719380 (+ 5299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:21.720627 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:21.720634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:21.722253 (+ 1619us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:21.723944 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:21.723978 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.725878 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.725883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:21.729260 (+ 3377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:21.729275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:21.731364 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:21.731377 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:21.731976 (+ 599us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:21.733202 (+ 1226us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:21.733249 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:21.737862 (+ 4613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:21.743231 (+ 5369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:21.743244 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:21.743266 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:21.744051 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:21.744682 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:21.744686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:21.744690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:21.744955 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:21.745780 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:21.745788 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:21.746308 (+ 520us) client_negotiation.cc:769] Sending connection context
0809 20:24:21.747970 (+ 1662us) client_negotiation.cc:240] Negotiation successful
0809 20:24:21.748575 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":429,"spinlock_wait_cycles":20224,"thread_start_us":723,"threads_started":1}
I20260809 20:24:21.756116 20121 heartbeater.cc:347] Connected to a master server at 127.12.78.60:38949
I20260809 20:24:21.756449 20121 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:21.756655 20122 heartbeater.cc:347] Connected to a master server at 127.12.78.62:39677
I20260809 20:24:21.756860 20122 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:21.757421 20123 heartbeater.cc:347] Connected to a master server at 127.12.78.61:39915
I20260809 20:24:21.757648 20123 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:21.758464 20122 heartbeater.cc:511] Master 127.12.78.62:39677 requested a full tablet report, sending...
I20260809 20:24:21.758529 20121 heartbeater.cc:511] Master 127.12.78.60:38949 requested a full tablet report, sending...
I20260809 20:24:21.758749 20123 heartbeater.cc:511] Master 127.12.78.61:39915 requested a full tablet report, sending...
I20260809 20:24:21.760731 19101 ts_manager.cc:194] Re-registered known tserver with Master: 24068fc4286f45a4bfe48c531efac812 (127.12.78.1:45283)
I20260809 20:24:21.760999 19732 ts_manager.cc:194] Registered new tserver with Master: 24068fc4286f45a4bfe48c531efac812 (127.12.78.1:45283)
I20260809 20:24:21.761310 19933 ts_manager.cc:194] Registered new tserver with Master: 24068fc4286f45a4bfe48c531efac812 (127.12.78.1:45283)
I20260809 20:24:21.762167 19101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:59485
I20260809 20:24:21.846050 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 19309
W20260809 20:24:21.864359 19085 connection.cc:535] server connection from 127.12.78.2:59911 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 20:24:21.865330 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:37183
--local_ip_for_outbound_sockets=127.12.78.2
--tserver_master_addrs=127.12.78.62:39677
--webserver_port=38533
--webserver_interface=127.12.78.2
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab
--principal=oryx/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.78.62:39677,127.12.78.61:39915
--tserver_master_addrs=127.12.78.62:39677,127.12.78.61:39915,127.12.78.60:38949 with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:24:22.024739 19887 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: NON_VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: true } } }
I20260809 20:24:22.025239 19887 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:22.027151 20134 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522: attempting to promote NON_VOTER 20eae6ac52e34db5abb5c061cf4051ca to VOTER
I20260809 20:24:22.027796 20134 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } }
I20260809 20:24:22.030220 19742 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 FOLLOWER]: Refusing update from remote peer ccd8c1717f7b4ce8b2489ede55f20522: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 20:24:22.030299 19944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 1 LEARNER]: Refusing update from remote peer ccd8c1717f7b4ce8b2489ede55f20522: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 20:24:22.030848 20134 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 20:24:22.031226 20135 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 20:24:22.034710 20134 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 20eae6ac52e34db5abb5c061cf4051ca (127.12.78.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } } }
I20260809 20:24:22.035465 19742 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 20eae6ac52e34db5abb5c061cf4051ca (127.12.78.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } } }
I20260809 20:24:22.038745 19887 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } } }
I20260809 20:24:22.039041 19887 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:22.039625 19887 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } } }
I20260809 20:24:22.040091 19887 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:22.040612 20134 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } } }
I20260809 20:24:22.041019 20134 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:22.040812 19944 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 20eae6ac52e34db5abb5c061cf4051ca (127.12.78.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } } }
I20260809 20:24:22.045220 19973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } } }
I20260809 20:24:22.045469 19973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:24:22.045282 20140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } } }
I20260809 20:24:22.045709 20140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 20:24:22.210366 20133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:22.210717 20133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:22.210901 20133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:22.218446 20133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:22.218613 20133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:22.218791 20133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:24:22.229400 20133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab
--principal=oryx/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:37183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=38533
--tserver_master_addrs=127.12.78.62:39677,127.12.78.61:39915,127.12.78.60:38949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:22.230816 20133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:22.232142 20133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:22.245767 20148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:22.245980 20145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:22.245952 20146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:22.246948 20133 server_base.cc:1034] running on GCE node
I20260809 20:24:22.247632 20133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:22.248687 20133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:22.249928 20133 hybrid_clock.cc:648] HybridClock initialized: now 1786307062249851 us; error 53 us; skew 500 ppm
I20260809 20:24:22.250250 20133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:22 dist-test-slave-rnrw krb5kdc[19054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307062, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:22.253461 20133 init.cc:377] Logged in from keytab as oryx/127.12.78.2@KRBTEST.COM (short username oryx)
I20260809 20:24:22.254918 20133 webserver.cc:460] Webserver started at http://127.12.78.2:38533/ using document root <none> and password file <none>
I20260809 20:24:22.255586 20133 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:22.255746 20133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:22.261085 20133 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 20:24:22.263830 20155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:22.265172 20133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:22.265349 20133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "fc594aa5090a47c5b511a917b5c7b24b"
format_stamp: "Formatted at 2026-08-09 20:24:18 on dist-test-slave-rnrw"
server_key: "538af637c5df1bff016c3ef1abff3927"
server_key_iv: "8b8d51619df12803090001b798a4185e"
server_key_version: "encryptionkey@0"
I20260809 20:24:22.266361 20133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:22.288645 20133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:22.294075 20133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:22.294369 20133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:22.296980 20133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:22.298575 20133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:22.298664 20133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:22.298797 20133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:22.298897 20133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:22.335426 20133 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:37183
I20260809 20:24:22.335469 20276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:37183 every 8 connection(s)
I20260809 20:24:22.337142 20133 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:24:22.339447 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 20133
I20260809 20:24:22.339784 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 19448
Aug 09 20:24:22 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307062, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.61@KRBTEST.COM
Aug 09 20:24:22 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307062, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
Aug 09 20:24:22 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.78.3@KRBTEST.COM
Aug 09 20:24:22 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307062, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.60@KRBTEST.COM
I20260809 20:24:22.383124 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.341170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:39915 (local address 127.12.78.2:55373)
0809 20:24:22.353621 (+ 12451us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:22.353670 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:22.358038 (+ 4368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:22.359093 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:22.359109 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:22.360196 (+ 1087us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:22.361410 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:22.361436 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.363442 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.363451 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:22.365855 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:22.365872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.367068 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.367073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:22.367274 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:22.368149 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:22.368178 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:22.372541 (+ 4363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:22.377439 (+ 4898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:22.377448 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:22.377464 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:22.378073 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:22.378559 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:22.378564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:22.378567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:22.378895 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:22.379757 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:22.379766 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:22.380223 (+ 457us) client_negotiation.cc:769] Sending connection context
0809 20:24:22.380703 (+ 480us) client_negotiation.cc:240] Negotiation successful
0809 20:24:22.381336 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12113,"mutex_wait_us":84,"thread_start_us":15393,"threads_started":1}
I20260809 20:24:22.383584 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.340590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:57291 (local address 127.12.78.62:39677)
0809 20:24:22.358992 (+ 18402us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:22.359000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:22.359034 (+ 34us) server_negotiation.cc:407] Connection header received
0809 20:24:22.359100 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:22.359106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:22.359185 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:22.359465 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:22.361665 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.362856 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.367157 (+ 4301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.367903 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.368058 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:22.374039 (+ 5981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:22.374044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:22.374048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:22.374080 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:22.379551 (+ 5471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:22.380754 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:22.380760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:22.380764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:22.380964 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:22.381501 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:22.381519 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:22.381524 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:22.381919 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:22.382061 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:22.382644 (+ 583us) server_negotiation.cc:299] Negotiation successful
0809 20:24:22.382894 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18250,"thread_start_us":18138,"threads_started":1}
I20260809 20:24:22.384647 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.340650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:55373 (local address 127.12.78.61:39915)
0809 20:24:22.357218 (+ 16568us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:22.357225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:22.357246 (+ 21us) server_negotiation.cc:407] Connection header received
0809 20:24:22.358194 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:22.358204 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:22.358299 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:22.358574 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:22.361677 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.363202 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.366024 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.366666 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.366741 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:22.372865 (+ 6124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:22.372872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:22.372876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:22.372919 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:22.377301 (+ 4382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:22.378244 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:22.378250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:22.378255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:22.378430 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:22.379037 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:22.379042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:22.379046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:22.379575 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:22.379793 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:22.383710 (+ 3917us) server_negotiation.cc:299] Negotiation successful
0809 20:24:22.383960 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16442,"thread_start_us":12170,"threads_started":1}
I20260809 20:24:22.385872 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.341670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.2:57291)
0809 20:24:22.355129 (+ 13459us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:22.355153 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:22.357653 (+ 2500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:22.359693 (+ 2040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:22.359699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:22.360344 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:22.361407 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:22.361433 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.363073 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.363082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:22.365855 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:22.365872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.368151 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.368157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:22.368393 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:22.369136 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:22.369163 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:22.373601 (+ 4438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:22.379714 (+ 6113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:22.379720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:22.379727 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:22.380612 (+ 885us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:22.381156 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:22.381163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:22.381168 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:22.381357 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:22.382153 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:22.382158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:22.382307 (+ 149us) client_negotiation.cc:769] Sending connection context
0809 20:24:22.384243 (+ 1936us) client_negotiation.cc:240] Negotiation successful
0809 20:24:22.384590 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13277,"mutex_wait_us":759,"thread_start_us":11501,"threads_started":1}
I20260809 20:24:22.386310 20280 heartbeater.cc:347] Connected to a master server at 127.12.78.61:39915
I20260809 20:24:22.386758 20280 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:22.387133 20279 heartbeater.cc:347] Connected to a master server at 127.12.78.62:39677
I20260809 20:24:22.387467 20279 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:22.388217 20280 heartbeater.cc:511] Master 127.12.78.61:39915 requested a full tablet report, sending...
I20260809 20:24:22.388339 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.341245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.12.78.3:43915)
0809 20:24:22.357714 (+ 16469us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:22.357724 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:22.357755 (+ 31us) server_negotiation.cc:407] Connection header received
0809 20:24:22.357841 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:22.357850 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:22.358137 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:22.358399 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:22.360729 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.362410 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.364942 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.366250 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.366418 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:22.376505 (+ 10087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:22.376519 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:22.376535 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:22.376593 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:22.383136 (+ 6543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:22.384202 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:22.384226 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:22.384230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:22.384523 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:22.385232 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:22.385239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:22.385242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:22.385794 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:22.386193 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:22.387639 (+ 1446us) server_negotiation.cc:299] Negotiation successful
0809 20:24:22.387924 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16274,"thread_start_us":16948,"threads_started":1}
I20260809 20:24:22.388463 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.354454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:48047 (local address 127.12.78.60:38949)
0809 20:24:22.364575 (+ 10121us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:22.364583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:22.364604 (+ 21us) server_negotiation.cc:407] Connection header received
0809 20:24:22.364665 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:22.364669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:22.364805 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:22.368338 (+ 3533us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:22.368618 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.370591 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.373162 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.374096 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.374224 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:22.378587 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:22.378592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:22.378598 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:22.378655 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:22.383113 (+ 4458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:22.384578 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:22.384583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:22.384587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:22.384813 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:22.385386 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:22.385392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:22.385395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:22.385870 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:22.386035 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:22.387501 (+ 1466us) server_negotiation.cc:299] Negotiation successful
0809 20:24:22.387774 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9932,"thread_start_us":9803,"threads_started":1}
I20260809 20:24:22.389055 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.354366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38949 (local address 127.12.78.2:48047)
0809 20:24:22.356967 (+ 2601us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:22.356986 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:22.357640 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:22.365042 (+ 7402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:22.365048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:22.366263 (+ 1215us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:22.366960 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:22.366975 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.370794 (+ 3819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.370815 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:22.372968 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:22.372983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.374524 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.374531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:22.374727 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:22.375517 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:22.375540 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:22.378336 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:22.383281 (+ 4945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:22.383289 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:22.383294 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:22.384417 (+ 1123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:22.384957 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:22.384961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:22.384964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:22.385214 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:22.386104 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:22.386108 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:22.386264 (+ 156us) client_negotiation.cc:769] Sending connection context
0809 20:24:22.386471 (+ 207us) client_negotiation.cc:240] Negotiation successful
0809 20:24:22.386764 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2457,"thread_start_us":340,"threads_started":1}
I20260809 20:24:22.389142 20279 heartbeater.cc:511] Master 127.12.78.62:39677 requested a full tablet report, sending...
I20260809 20:24:22.390061 19576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:22.390202 20278 heartbeater.cc:347] Connected to a master server at 127.12.78.60:38949
I20260809 20:24:22.390321 20278 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:22.390750 19101 ts_manager.cc:194] Re-registered known tserver with Master: fc594aa5090a47c5b511a917b5c7b24b (127.12.78.2:37183)
I20260809 20:24:22.390856 19732 ts_manager.cc:194] Registered new tserver with Master: fc594aa5090a47c5b511a917b5c7b24b (127.12.78.2:37183)
I20260809 20:24:22.390964 20278 heartbeater.cc:511] Master 127.12.78.60:38949 requested a full tablet report, sending...
I20260809 20:24:22.392253 19101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:57291
I20260809 20:24:22.392711 19933 ts_manager.cc:194] Registered new tserver with Master: fc594aa5090a47c5b511a917b5c7b24b (127.12.78.2:37183)
I20260809 20:24:22.478050 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 19448
I20260809 20:24:22.496793 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:43915
--local_ip_for_outbound_sockets=127.12.78.3
--tserver_master_addrs=127.12.78.62:39677
--webserver_port=41219
--webserver_interface=127.12.78.3
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab
--principal=oryx/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.78.62:39677,127.12.78.61:39915
--tserver_master_addrs=127.12.78.62:39677,127.12.78.61:39915,127.12.78.60:38949 with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:24:22.765090 20122 heartbeater.cc:503] Master 127.12.78.62:39677 was elected leader, sending a full tablet report...
W20260809 20:24:22.836653 20291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:22.836937 20291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:22.837035 20291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:22.844429 20291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:22.844588 20291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:22.844733 20291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:24:22.854746 20291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab
--principal=oryx/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:43915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=41219
--tserver_master_addrs=127.12.78.62:39677,127.12.78.61:39915,127.12.78.60:38949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:22.856057 20291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:22.857451 20291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:22.871104 20300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:22.871205 20298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:22.871179 20297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:22.872502 20291 server_base.cc:1034] running on GCE node
I20260809 20:24:22.873148 20291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:22.874027 20291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:22.875259 20291 hybrid_clock.cc:648] HybridClock initialized: now 1786307062875197 us; error 56 us; skew 500 ppm
I20260809 20:24:22.875666 20291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:22 dist-test-slave-rnrw krb5kdc[19054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307062, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:22.879173 20291 init.cc:377] Logged in from keytab as oryx/127.12.78.3@KRBTEST.COM (short username oryx)
I20260809 20:24:22.880761 20291 webserver.cc:460] Webserver started at http://127.12.78.3:41219/ using document root <none> and password file <none>
I20260809 20:24:22.881546 20291 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:22.881666 20291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:22.887069 20291 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.003s
I20260809 20:24:22.890041 20307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:22.891515 20291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:22.891680 20291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "5d4d517e110a40bb9f5f3caae6e3e607"
format_stamp: "Formatted at 2026-08-09 20:24:19 on dist-test-slave-rnrw"
server_key: "80c87741cd110bd382ca6e0bef85921f"
server_key_iv: "5ecc0f18ad849385e5f38f40a6f252e7"
server_key_version: "encryptionkey@0"
I20260809 20:24:22.892724 20291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:22.910172 20291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:22.914619 20291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:22.914894 20291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:22.915930 20291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:22.917374 20291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:22.917503 20291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:22.917656 20291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:22.917753 20291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:22.952425 20291 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:43915
I20260809 20:24:22.952526 20428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:43915 every 8 connection(s)
I20260809 20:24:22.953974 20291 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:24:22.961856 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 20291
Aug 09 20:24:23 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307062, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
Aug 09 20:24:23 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307062, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.61@KRBTEST.COM
Aug 09 20:24:23 dist-test-slave-rnrw krb5kdc[19054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307062, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.60@KRBTEST.COM
I20260809 20:24:23.028076 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.956898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:38247 (local address 127.12.78.62:39677)
0809 20:24:22.987326 (+ 30428us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:22.987335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:22.987363 (+ 28us) server_negotiation.cc:407] Connection header received
0809 20:24:22.989755 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:22.989762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:22.989848 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:22.990003 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:22.993572 (+ 3569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.995100 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.001925 (+ 6825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.002798 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.002941 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.015403 (+ 12462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:23.015411 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:23.015415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:23.015459 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:23.020783 (+ 5324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.023211 (+ 2428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.023217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.023221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.023797 (+ 576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.024792 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.024810 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.024813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.025636 (+ 823us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:23.025845 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.027138 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.027453 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30269,"thread_start_us":30211,"threads_started":1}
I20260809 20:24:23.030481 20447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.989883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.12.78.61:39915)
0809 20:24:22.995416 (+ 5533us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:22.995425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:22.997647 (+ 2222us) server_negotiation.cc:407] Connection header received
0809 20:24:23.000190 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.000196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.000281 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.000457 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:23.004890 (+ 4433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.006548 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.009141 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.010196 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.010337 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.014875 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:23.014883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:23.014889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:23.014933 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:23.025501 (+ 10568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.027274 (+ 1773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.027280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.027283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.027514 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.027989 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.027997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.028003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.028377 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:23.029717 (+ 1340us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.029851 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.030076 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3490,"thread_start_us":3305,"threads_started":1}
I20260809 20:24:23.030820 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.989585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.12.78.62:39677)
0809 20:24:22.996950 (+ 7365us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:22.996955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:22.996980 (+ 25us) server_negotiation.cc:407] Connection header received
0809 20:24:22.997036 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:22.997042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:22.997138 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:22.997423 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:23.000720 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.003227 (+ 2507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.007852 (+ 4625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.008691 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.008843 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.017138 (+ 8295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:23.017148 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:23.017155 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:23.017198 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:23.026360 (+ 9162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.027811 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.027816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.027820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.027965 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.028412 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.028474 (+ 62us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.028480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.028882 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:23.030046 (+ 1164us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.030177 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.030395 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7220,"thread_start_us":7071,"threads_started":1}
I20260809 20:24:23.031472 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.957558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.3:38247)
0809 20:24:22.986829 (+ 29271us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:22.986856 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:22.989590 (+ 2734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:22.990856 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:22.990870 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:22.992841 (+ 1971us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:22.993331 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:22.993366 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.000208 (+ 6842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.000213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.001776 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.001790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.005083 (+ 3293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.005094 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.005564 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.006872 (+ 1308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:23.006914 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:23.015047 (+ 8133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:23.022139 (+ 7092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:23.022149 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:23.022170 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:23.023037 (+ 867us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:23.024018 (+ 981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:23.024023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:23.024027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:23.024596 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:23.025961 (+ 1365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:23.025972 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:23.026615 (+ 643us) client_negotiation.cc:769] Sending connection context
0809 20:24:23.028851 (+ 2236us) client_negotiation.cc:240] Negotiation successful
0809 20:24:23.029532 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28314,"mutex_wait_us":39,"spinlock_wait_cycles":38272,"thread_start_us":10842,"threads_started":1}
I20260809 20:24:23.033598 20431 heartbeater.cc:347] Connected to a master server at 127.12.78.62:39677
I20260809 20:24:23.033844 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.976306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38949 (local address 127.12.78.3:57367)
0809 20:24:22.988206 (+ 11900us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:22.988224 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:22.991266 (+ 3042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:22.996441 (+ 5175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:22.996449 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:23.005219 (+ 8770us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:23.010149 (+ 4930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.010165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.012510 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.012516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.014711 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.014727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.015991 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.016000 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.016194 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.018010 (+ 1816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:23.018035 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:23.021094 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:23.026753 (+ 5659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:23.026757 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:23.026762 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:23.027378 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:23.028256 (+ 878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:23.028261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:23.028265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:23.028504 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:23.031554 (+ 3050us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:23.031559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:23.031698 (+ 139us) client_negotiation.cc:769] Sending connection context
0809 20:24:23.031904 (+ 206us) client_negotiation.cc:240] Negotiation successful
0809 20:24:23.033198 (+ 1294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11761,"mutex_wait_us":464,"thread_start_us":9670,"threads_started":1}
I20260809 20:24:23.033988 20431 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:23.034931 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.970023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:57367 (local address 127.12.78.60:38949)
0809 20:24:22.987598 (+ 17575us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:22.987607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:22.988418 (+ 811us) server_negotiation.cc:407] Connection header received
0809 20:24:22.991499 (+ 3081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:22.991507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:22.991630 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:22.991934 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:23.010336 (+ 18402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.012226 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.014901 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.015579 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.015712 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.021331 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:23.021338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:23.021343 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:23.021390 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:23.026597 (+ 5207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.027850 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.027867 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.027870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.028098 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.029475 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.029480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.029484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.031244 (+ 1760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:23.031473 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.034043 (+ 2570us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.034347 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":17393,"thread_start_us":17585,"threads_started":1}
I20260809 20:24:23.035081 20431 heartbeater.cc:511] Master 127.12.78.62:39677 requested a full tablet report, sending...
I20260809 20:24:23.035732 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.956888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:51533 (local address 127.12.78.61:39915)
0809 20:24:22.986780 (+ 29892us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:22.986789 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:22.986818 (+ 29us) server_negotiation.cc:407] Connection header received
0809 20:24:22.990430 (+ 3612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:22.990440 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:22.990629 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:22.990825 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:22.993767 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.995268 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.000214 (+ 4946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.001242 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.001410 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.016132 (+ 14722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:23.016138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:23.016143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:23.016191 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:23.020527 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.028670 (+ 8143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.028678 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.028682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.028885 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.030167 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.030174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.030178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.030595 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:23.030722 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.035036 (+ 4314us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.035265 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":29746,"thread_start_us":29639,"threads_started":1}
I20260809 20:24:23.036751 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.957558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:39915 (local address 127.12.78.3:51533)
0809 20:24:22.980846 (+ 23288us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:22.980939 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:22.989335 (+ 8396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:22.991056 (+ 1721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:22.991064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:22.991999 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:22.993570 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:22.993583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:22.995548 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:22.995569 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.000006 (+ 4437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.000037 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.010413 (+ 10376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.010419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.010669 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.011974 (+ 1305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:23.012007 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:23.015835 (+ 3828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:23.026524 (+ 10689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:23.026532 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:23.026537 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:23.027704 (+ 1167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:23.029027 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:23.029031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:23.029034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:23.029287 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:23.030803 (+ 1516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:23.030808 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:23.030947 (+ 139us) client_negotiation.cc:769] Sending connection context
0809 20:24:23.033219 (+ 2272us) client_negotiation.cc:240] Negotiation successful
0809 20:24:23.033636 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22815,"mutex_wait_us":510,"thread_start_us":30229,"threads_started":1}
I20260809 20:24:23.036913 19102 ts_manager.cc:194] Re-registered known tserver with Master: 5d4d517e110a40bb9f5f3caae6e3e607 (127.12.78.3:43915)
I20260809 20:24:23.037858 20433 heartbeater.cc:347] Connected to a master server at 127.12.78.61:39915
I20260809 20:24:23.038031 20433 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:23.038411 19102 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:38247
I20260809 20:24:23.038605 20433 heartbeater.cc:511] Master 127.12.78.61:39915 requested a full tablet report, sending...
I20260809 20:24:23.040340 19732 ts_manager.cc:194] Registered new tserver with Master: 5d4d517e110a40bb9f5f3caae6e3e607 (127.12.78.3:43915)
I20260809 20:24:23.040824 20429 heartbeater.cc:347] Connected to a master server at 127.12.78.60:38949
I20260809 20:24:23.040992 20429 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:23.042523 20429 heartbeater.cc:511] Master 127.12.78.60:38949 requested a full tablet report, sending...
I20260809 20:24:23.044077 19933 ts_manager.cc:194] Registered new tserver with Master: 5d4d517e110a40bb9f5f3caae6e3e607 (127.12.78.3:43915)
I20260809 20:24:23.047151 20449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:22.997484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.12.78.60:38949)
0809 20:24:23.006604 (+ 9120us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:23.006621 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:23.009775 (+ 3154us) server_negotiation.cc:407] Connection header received
0809 20:24:23.009954 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.009959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.010046 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.010177 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:23.020404 (+ 10227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.022468 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.025068 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.025956 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.026081 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.038073 (+ 11992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:23.038080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:23.038085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:23.038123 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:23.043690 (+ 5567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.044661 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.044665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.044668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.044835 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.045329 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.045335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.045339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.045762 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:23.046047 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.046392 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.046661 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":8955,"thread_start_us":12343,"threads_started":1}
I20260809 20:24:23.035955 19101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:24:23.049518 19101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:24:23.095427 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.071913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:45283 (local address 127.0.0.1:35806)
0809 20:24:23.083218 (+ 11305us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:23.083245 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:23.083533 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:23.086683 (+ 3150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:23.086691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:23.086731 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:23.087324 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.087341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.089463 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.089473 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.092730 (+ 3257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.092755 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.094454 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.094460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.094652 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.094688 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 20:24:23.094803 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 20:24:23.094996 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11055,"thread_start_us":11729,"threads_started":1}
I20260809 20:24:23.096952 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.072328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35806 (local address 127.12.78.1:45283)
0809 20:24:23.085738 (+ 13410us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:23.085747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:23.085776 (+ 29us) server_negotiation.cc:407] Connection header received
0809 20:24:23.085895 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.085907 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.086278 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.086662 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:23.087599 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.089267 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.092957 (+ 3690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.094255 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.094413 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.094665 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.095786 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.095981 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13075,"thread_start_us":17781,"threads_started":1}
I20260809 20:24:23.098834 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.072226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:37183 (local address 127.0.0.1:49988)
0809 20:24:23.085268 (+ 13042us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:23.085287 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:23.085563 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:23.087894 (+ 2331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:23.087902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:23.087939 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:23.088734 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.088753 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.091691 (+ 2938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.091700 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.095390 (+ 3690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.095419 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.098031 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.098038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.098227 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.098251 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 20:24:23.098352 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 20:24:23.098481 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12854,"thread_start_us":12481,"threads_started":1}
I20260809 20:24:23.099061 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.072492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.12.78.2:37183)
0809 20:24:23.082735 (+ 10243us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:23.082746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:23.087024 (+ 4278us) server_negotiation.cc:407] Connection header received
0809 20:24:23.087153 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.087165 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.087430 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.089004 (+ 1574us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:23.089379 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.091493 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.095853 (+ 4360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.097774 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.097910 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.098147 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.098410 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.098621 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9908,"thread_start_us":9688,"threads_started":1}
I20260809 20:24:23.100811 20050 tablet_service.cc:1467] Processing CreateTablet for tablet 9435a03b35894609bae5fadb6ca41462 (DEFAULT_TABLE table=test-table [id=78a17afd6f184e68b2d51bce1791acf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:23.102653 20050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9435a03b35894609bae5fadb6ca41462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:23.103161 20206 tablet_service.cc:1467] Processing CreateTablet for tablet 9435a03b35894609bae5fadb6ca41462 (DEFAULT_TABLE table=test-table [id=78a17afd6f184e68b2d51bce1791acf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:23.104736 20206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9435a03b35894609bae5fadb6ca41462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:23.106266 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.084057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:43915 (local address 127.0.0.1:55216)
0809 20:24:23.095466 (+ 11409us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:23.095487 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:23.095772 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:23.097461 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:23.097465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:23.097491 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:23.097863 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.097875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.100972 (+ 3097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.100980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.103351 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.103364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.105248 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.105255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.105506 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.105532 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 20:24:23.105638 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 20:24:23.105772 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11172,"thread_start_us":10974,"threads_started":1}
I20260809 20:24:23.106629 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.084753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.12.78.3:43915)
0809 20:24:23.096635 (+ 11882us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:23.096644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:23.096677 (+ 33us) server_negotiation.cc:407] Connection header received
0809 20:24:23.096824 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.096833 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.097148 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.097464 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:23.099246 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.100840 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.103554 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.105055 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.105204 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.105451 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.105706 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.105905 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11681,"thread_start_us":11480,"threads_started":1}
I20260809 20:24:23.110489 20361 tablet_service.cc:1467] Processing CreateTablet for tablet 9435a03b35894609bae5fadb6ca41462 (DEFAULT_TABLE table=test-table [id=78a17afd6f184e68b2d51bce1791acf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:23.111676 20361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9435a03b35894609bae5fadb6ca41462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:23.122254 20469 tablet_bootstrap.cc:492] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812: Bootstrap starting.
I20260809 20:24:23.127097 20471 tablet_bootstrap.cc:492] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607: Bootstrap starting.
I20260809 20:24:23.129164 20469 tablet_bootstrap.cc:654] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:23.129297 20470 tablet_bootstrap.cc:492] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b: Bootstrap starting.
I20260809 20:24:23.130782 20469 log.cc:824] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:23.132403 20471 tablet_bootstrap.cc:654] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:23.134203 20471 log.cc:824] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:23.135046 20470 tablet_bootstrap.cc:654] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:23.135103 20469 tablet_bootstrap.cc:492] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812: No bootstrap required, opened a new log
I20260809 20:24:23.135459 20469 ts_tablet_manager.cc:1397] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260809 20:24:23.136648 20470 log.cc:824] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:23.137616 20471 tablet_bootstrap.cc:492] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607: No bootstrap required, opened a new log
I20260809 20:24:23.137972 20471 ts_tablet_manager.cc:1397] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260809 20:24:23.141788 20470 tablet_bootstrap.cc:492] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b: No bootstrap required, opened a new log
I20260809 20:24:23.142160 20470 ts_tablet_manager.cc:1397] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260809 20:24:23.150228 20469 raft_consensus.cc:348] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:23.150923 20469 raft_consensus.cc:374] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:23.151149 20469 raft_consensus.cc:729] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24068fc4286f45a4bfe48c531efac812, State: Initialized, Role: FOLLOWER
I20260809 20:24:23.151190 20471 raft_consensus.cc:348] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:23.152129 20471 raft_consensus.cc:374] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:23.152134 20469 consensus_queue.cc:260] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:23.152318 20471 raft_consensus.cc:729] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4d517e110a40bb9f5f3caae6e3e607, State: Initialized, Role: FOLLOWER
I20260809 20:24:23.153285 20471 consensus_queue.cc:260] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:23.156186 20470 raft_consensus.cc:348] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:23.157035 20470 raft_consensus.cc:374] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:23.157284 20470 raft_consensus.cc:729] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc594aa5090a47c5b511a917b5c7b24b, State: Initialized, Role: FOLLOWER
I20260809 20:24:23.158447 20470 consensus_queue.cc:260] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:23.161332 20469 ts_tablet_manager.cc:1428] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812: Time spent starting tablet: real 0.026s user 0.009s sys 0.017s
I20260809 20:24:23.162139 20471 ts_tablet_manager.cc:1428] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607: Time spent starting tablet: real 0.024s user 0.009s sys 0.012s
I20260809 20:24:23.167346 20470 ts_tablet_manager.cc:1428] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b: Time spent starting tablet: real 0.025s user 0.010s sys 0.009s
I20260809 20:24:23.167690 20279 heartbeater.cc:503] Master 127.12.78.62:39677 was elected leader, sending a full tablet report...
I20260809 20:24:23.169411 20431 heartbeater.cc:503] Master 127.12.78.62:39677 was elected leader, sending a full tablet report...
I20260809 20:24:23.179628 20477 raft_consensus.cc:482] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:24:23.180024 20477 raft_consensus.cc:504] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:23.181937 20477 leader_election.cc:288] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24068fc4286f45a4bfe48c531efac812 (127.12.78.1:45283), 5d4d517e110a40bb9f5f3caae6e3e607 (127.12.78.3:43915)
I20260809 20:24:23.190052 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.182610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:41685 (local address 127.12.78.1:45283)
0809 20:24:23.182867 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:23.182873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:23.183408 (+ 535us) server_negotiation.cc:407] Connection header received
0809 20:24:23.183689 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.183694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.183755 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.183852 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:23.184461 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.185444 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.187868 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.188793 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.188963 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.189076 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.189486 (+ 410us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.189627 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 20:24:23.190184 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.182177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:45283 (local address 127.12.78.2:41685)
0809 20:24:23.183086 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:23.183107 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:23.183537 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:23.183868 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:23.183872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:23.183895 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:23.184274 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.184286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.185616 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.185620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.187595 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.187625 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.189067 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.189073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.189297 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.189322 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 20:24:23.189446 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 20:24:23.189589 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":20,"thread_start_us":555,"threads_started":1}
I20260809 20:24:23.191412 20476 raft_consensus.cc:482] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:24:23.191378 20072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9435a03b35894609bae5fadb6ca41462" candidate_uuid: "fc594aa5090a47c5b511a917b5c7b24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24068fc4286f45a4bfe48c531efac812" is_pre_election: true
I20260809 20:24:23.191944 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.183380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:60897 (local address 127.12.78.3:43915)
0809 20:24:23.183573 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:23.183579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:23.185042 (+ 1463us) server_negotiation.cc:407] Connection header received
0809 20:24:23.185231 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.185236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.185322 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.185407 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:23.186387 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.187596 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.189806 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.190899 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.191055 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.191160 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.191482 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.191623 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 20:24:23.192222 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.183116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:43915 (local address 127.12.78.2:60897)
0809 20:24:23.184872 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:23.184887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:23.185102 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:23.185422 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:23.185426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:23.185448 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:23.186141 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.186151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.187737 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.187744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.189637 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.189650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.191119 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.191126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.191315 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.191338 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 20:24:23.191440 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 20:24:23.191564 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1649,"mutex_wait_us":1,"thread_start_us":1391,"threads_started":1}
I20260809 20:24:23.191920 20476 raft_consensus.cc:504] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:23.192253 20072 raft_consensus.cc:2454] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc594aa5090a47c5b511a917b5c7b24b in term 0.
I20260809 20:24:23.193483 20158 leader_election.cc:302] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24068fc4286f45a4bfe48c531efac812, fc594aa5090a47c5b511a917b5c7b24b; no voters:
I20260809 20:24:23.193485 20381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9435a03b35894609bae5fadb6ca41462" candidate_uuid: "fc594aa5090a47c5b511a917b5c7b24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" is_pre_election: true
I20260809 20:24:23.194094 20381 raft_consensus.cc:2454] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc594aa5090a47c5b511a917b5c7b24b in term 0.
I20260809 20:24:23.194289 20477 raft_consensus.cc:2781] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:24:23.194473 20477 raft_consensus.cc:482] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:24:23.194597 20477 raft_consensus.cc:3037] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:23.196002 20476 leader_election.cc:288] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24068fc4286f45a4bfe48c531efac812 (127.12.78.1:45283), fc594aa5090a47c5b511a917b5c7b24b (127.12.78.2:37183)
I20260809 20:24:23.198122 20477 raft_consensus.cc:504] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:23.199055 20477 leader_election.cc:288] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [CANDIDATE]: Term 1 election: Requested vote from peers 24068fc4286f45a4bfe48c531efac812 (127.12.78.1:45283), 5d4d517e110a40bb9f5f3caae6e3e607 (127.12.78.3:43915)
I20260809 20:24:23.199529 20072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9435a03b35894609bae5fadb6ca41462" candidate_uuid: "fc594aa5090a47c5b511a917b5c7b24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24068fc4286f45a4bfe48c531efac812"
I20260809 20:24:23.199761 20072 raft_consensus.cc:3037] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:23.202557 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.194543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:37183 (local address 127.12.78.3:56029)
0809 20:24:23.195403 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:23.195421 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:23.195538 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:23.196032 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:23.196037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:23.196062 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:23.196299 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.196313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.197885 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.197893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.200019 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.200032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.201697 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.201703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.201956 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.201980 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 20:24:23.202099 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 20:24:23.202244 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":705}
I20260809 20:24:23.202595 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.194757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:56029 (local address 127.12.78.2:37183)
0809 20:24:23.195177 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:23.195181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:23.195510 (+ 329us) server_negotiation.cc:407] Connection header received
0809 20:24:23.195655 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.195661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.195736 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.195839 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:23.196460 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.197731 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.200227 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.201532 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.201684 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.201777 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.202113 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.202244 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260809 20:24:23.203692 20228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9435a03b35894609bae5fadb6ca41462" candidate_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc594aa5090a47c5b511a917b5c7b24b" is_pre_election: true
I20260809 20:24:23.203974 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.194517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:45283 (local address 127.12.78.3:56855)
0809 20:24:23.194874 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:23.194894 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:23.195008 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:23.195794 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:23.195798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:23.195822 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:23.196197 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.196218 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.198032 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.198036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.200595 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:23.200610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.202913 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.202920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:23.203230 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.203273 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 20:24:23.203386 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 20:24:23.203544 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":2}
I20260809 20:24:23.204046 20381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9435a03b35894609bae5fadb6ca41462" candidate_uuid: "fc594aa5090a47c5b511a917b5c7b24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4d517e110a40bb9f5f3caae6e3e607"
I20260809 20:24:23.204381 20228 raft_consensus.cc:2379] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d4d517e110a40bb9f5f3caae6e3e607 in current term 1: Already voted for candidate fc594aa5090a47c5b511a917b5c7b24b in this term.
I20260809 20:24:23.204336 20381 raft_consensus.cc:3037] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:23.205296 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.194942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:56855 (local address 127.12.78.1:45283)
0809 20:24:23.195505 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:23.195511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:23.195528 (+ 17us) server_negotiation.cc:407] Connection header received
0809 20:24:23.195587 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.195593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.195670 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.195764 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:23.196369 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.197897 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.200811 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.202671 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.204439 (+ 1768us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.204589 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.204646 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.204836 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427}
I20260809 20:24:23.205878 20072 raft_consensus.cc:2454] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc594aa5090a47c5b511a917b5c7b24b in term 1.
I20260809 20:24:23.206226 20071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9435a03b35894609bae5fadb6ca41462" candidate_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24068fc4286f45a4bfe48c531efac812" is_pre_election: true
I20260809 20:24:23.206965 20071 raft_consensus.cc:2379] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d4d517e110a40bb9f5f3caae6e3e607 in current term 1: Already voted for candidate fc594aa5090a47c5b511a917b5c7b24b in this term.
W20260809 20:24:23.207005 20434 tablet.cc:2369] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:24:23.208057 20310 leader_election.cc:302] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d4d517e110a40bb9f5f3caae6e3e607; no voters: 24068fc4286f45a4bfe48c531efac812, fc594aa5090a47c5b511a917b5c7b24b
I20260809 20:24:23.208731 20381 raft_consensus.cc:2454] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc594aa5090a47c5b511a917b5c7b24b in term 1.
I20260809 20:24:23.208899 20158 leader_election.cc:302] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24068fc4286f45a4bfe48c531efac812, fc594aa5090a47c5b511a917b5c7b24b; no voters:
I20260809 20:24:23.208977 20476 raft_consensus.cc:2726] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 20:24:23.209297 20477 raft_consensus.cc:2781] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:23.209894 20477 raft_consensus.cc:686] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 1 LEADER]: Becoming Leader. State: Replica: fc594aa5090a47c5b511a917b5c7b24b, State: Running, Role: LEADER
I20260809 20:24:23.210443 20477 consensus_queue.cc:237] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
W20260809 20:24:23.212932 20124 tablet.cc:2369] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:24:23.216061 19101 catalog_manager.cc:5495] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b reported cstate change: term changed from 0 to 1, leader changed from <none> to fc594aa5090a47c5b511a917b5c7b24b (127.12.78.2). New cstate: current_term: 1 leader_uuid: "fc594aa5090a47c5b511a917b5c7b24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } health_report { overall_health: UNKNOWN } } }
I20260809 20:24:23.257637 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.251395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50002 (local address 127.12.78.2:37183)
0809 20:24:23.251614 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:23.251619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:23.251636 (+ 17us) server_negotiation.cc:407] Connection header received
0809 20:24:23.251727 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.251732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.251826 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.252046 (+ 220us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:24:23.252493 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.253463 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.255099 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.255619 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.255775 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.256326 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:24:23.256976 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:24:23.257160 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.257223 (+ 63us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.257313 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 20:24:23.267643 20381 raft_consensus.cc:1261] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [term 1 FOLLOWER]: Refusing update from remote peer fc594aa5090a47c5b511a917b5c7b24b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:23.267653 20071 raft_consensus.cc:1261] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812 [term 1 FOLLOWER]: Refusing update from remote peer fc594aa5090a47c5b511a917b5c7b24b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:23.268760 20480 consensus_queue.cc:1035] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:23.269342 20477 consensus_queue.cc:1035] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [LEADER]: Connected to new peer: Peer: permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:23.283166 20483 mvcc.cc:205] Tried to move back new op lower bound from 7316713731132440576 to 7316713730914578432. Current Snapshot: MvccSnapshot[applied={T|T < 7316713731132440576}]
I20260809 20:24:23.285386 20485 mvcc.cc:205] Tried to move back new op lower bound from 7316713731132440576 to 7316713730914578432. Current Snapshot: MvccSnapshot[applied={T|T < 7316713731132440576}]
I20260809 20:24:23.327100 19101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:23.327525 19101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:23.334818 19101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccd8c1717f7b4ce8b2489ede55f20522: Sending DeleteTablet for 3 replicas of tablet 9435a03b35894609bae5fadb6ca41462
I20260809 20:24:23.336058 20050 tablet_service.cc:1514] Processing DeleteTablet for tablet 9435a03b35894609bae5fadb6ca41462 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:23 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:35806
I20260809 20:24:23.336656 20361 tablet_service.cc:1514] Processing DeleteTablet for tablet 9435a03b35894609bae5fadb6ca41462 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:23 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:55216
I20260809 20:24:23.337327 20206 tablet_service.cc:1514] Processing DeleteTablet for tablet 9435a03b35894609bae5fadb6ca41462 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:23 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:49988
I20260809 20:24:23.357635 20497 tablet_replica.cc:330] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b: stopping tablet replica
I20260809 20:24:23.358712 20497 raft_consensus.cc:2229] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:24:23.359797 20497 raft_consensus.cc:2258] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:23.362582 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 19068
I20260809 20:24:23.363441 20497 ts_tablet_manager.cc:1905] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:23.364348 19132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:23.369382 20493 tablet_replica.cc:330] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812: stopping tablet replica
I20260809 20:24:23.369973 20493 raft_consensus.cc:2229] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:23.370545 20493 raft_consensus.cc:2258] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:23.374750 20493 ts_tablet_manager.cc:1905] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:23.383131 20494 tablet_replica.cc:330] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607: stopping tablet replica
I20260809 20:24:23.383375 20493 ts_tablet_manager.cc:1918] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:23.383594 20493 log.cc:1196] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/9435a03b35894609bae5fadb6ca41462
I20260809 20:24:23.384984 20493 ts_tablet_manager.cc:1939] T 9435a03b35894609bae5fadb6ca41462 P 24068fc4286f45a4bfe48c531efac812: Deleting consensus metadata
I20260809 20:24:23.385895 20494 raft_consensus.cc:2229] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:23.386160 20497 ts_tablet_manager.cc:1918] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:23.386521 20497 log.cc:1196] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/9435a03b35894609bae5fadb6ca41462
I20260809 20:24:23.386538 20494 raft_consensus.cc:2258] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:23.388314 20497 ts_tablet_manager.cc:1939] T 9435a03b35894609bae5fadb6ca41462 P fc594aa5090a47c5b511a917b5c7b24b: Deleting consensus metadata
I20260809 20:24:23.392158 20494 ts_tablet_manager.cc:1905] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:23.407164 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.344736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.12.78.61:39915)
0809 20:24:23.345002 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:23.345011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:23.360246 (+ 15235us) server_negotiation.cc:407] Connection header received
0809 20:24:23.360609 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.360623 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.360713 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.360815 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:23.377491 (+ 16676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.379145 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.382111 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.383243 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.383375 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.389866 (+ 6491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:23.389876 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:23.389882 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:23.389973 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:23.394298 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.396636 (+ 2338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.396642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.396646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.396884 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.397318 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.397326 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.397332 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.397766 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:23.398100 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.406393 (+ 8293us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.406713 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 20:24:23.407814 20494 ts_tablet_manager.cc:1918] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:23.408079 20494 log.cc:1196] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/9435a03b35894609bae5fadb6ca41462
I20260809 20:24:23.412978 20494 ts_tablet_manager.cc:1939] T 9435a03b35894609bae5fadb6ca41462 P 5d4d517e110a40bb9f5f3caae6e3e607: Deleting consensus metadata
I20260809 20:24:23.428505 20449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.361436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.12.78.60:38949)
0809 20:24:23.361705 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:23.361713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:23.399281 (+ 37568us) server_negotiation.cc:407] Connection header received
0809 20:24:23.399597 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.399604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.399711 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.399789 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:23.408915 (+ 9126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.410283 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.412515 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.413419 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.413626 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.418003 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:23.418009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:23.418015 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:23.418059 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:23.423063 (+ 5004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.424143 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.424149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.424153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.424363 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.425159 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.425166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.425169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.425894 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:23.426315 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.427727 (+ 1412us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.428057 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 20:24:23.551498 19087 catalog_manager.cc:4841] TS 5d4d517e110a40bb9f5f3caae6e3e607 (127.12.78.3:43915): tablet 9435a03b35894609bae5fadb6ca41462 (table test-table [id=78a17afd6f184e68b2d51bce1791acf4]) successfully deleted
I20260809 20:24:23.551990 19085 catalog_manager.cc:4841] TS fc594aa5090a47c5b511a917b5c7b24b (127.12.78.2:37183): tablet 9435a03b35894609bae5fadb6ca41462 (table test-table [id=78a17afd6f184e68b2d51bce1791acf4]) successfully deleted
I20260809 20:24:23.553140 19087 catalog_manager.cc:4841] TS 24068fc4286f45a4bfe48c531efac812 (127.12.78.1:45283): tablet 9435a03b35894609bae5fadb6ca41462 (table test-table [id=78a17afd6f184e68b2d51bce1791acf4]) successfully deleted
I20260809 20:24:23.553649 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 19068
I20260809 20:24:23.605928 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.585126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.12.78.61:39915)
0809 20:24:23.585314 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:23.585322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:23.586199 (+ 877us) server_negotiation.cc:407] Connection header received
0809 20:24:23.586524 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.586533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.586655 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.586749 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:23.589741 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.591289 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.594190 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.595097 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.595251 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.599713 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:23.599718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:23.599721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:23.599756 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:23.603134 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.603779 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.603785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.603789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.603915 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.604368 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.604373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.604377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.604792 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:23.605083 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.605336 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.605558 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 20:24:23.612841 20449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:23.585836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.12.78.60:38949)
0809 20:24:23.590842 (+ 5006us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:23.590848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:23.596975 (+ 6127us) server_negotiation.cc:407] Connection header received
0809 20:24:23.597296 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:23.597323 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:23.597416 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:23.597552 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:23.599515 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.600671 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.602545 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:23.603215 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:23.603338 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:23.607683 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:23.607687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:23.607690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:23.607724 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:23.610261 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.611004 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.611007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.611010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.611144 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:23.611506 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:23.611508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:23.611510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:23.611787 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:23.611958 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:23.612331 (+ 373us) server_negotiation.cc:299] Negotiation successful
0809 20:24:23.612488 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4855}
I20260809 20:24:24.306428 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.305606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.3:34705)
0809 20:24:24.306031 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.306159 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":213,"threads_started":1}
W20260809 20:24:24.306993 20431 heartbeater.cc:644] Failed to heartbeat to 127.12.78.62:39677 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
I20260809 20:24:24.321066 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.320338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.2:53181)
0809 20:24:24.320788 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.320894 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":259,"threads_started":1}
W20260809 20:24:24.321458 20279 heartbeater.cc:644] Failed to heartbeat to 127.12.78.62:39677 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
I20260809 20:24:24.325767 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.324970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.1:37733)
0809 20:24:24.325473 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.325588 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":231,"threads_started":1}
W20260809 20:24:24.326196 20122 heartbeater.cc:644] Failed to heartbeat to 127.12.78.62:39677 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
I20260809 20:24:24.834785 20509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ccd8c1717f7b4ce8b2489ede55f20522)
I20260809 20:24:24.835137 20509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } }
I20260809 20:24:24.835906 20510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ccd8c1717f7b4ce8b2489ede55f20522)
I20260809 20:24:24.836164 20510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } }
I20260809 20:24:24.836854 20509 leader_election.cc:288] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ccd8c1717f7b4ce8b2489ede55f20522 (127.12.78.62:39677), 45dcfa89acf6426c930dddd0e37f1254 (127.12.78.61:39915)
I20260809 20:24:24.837658 20510 leader_election.cc:288] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ccd8c1717f7b4ce8b2489ede55f20522 (127.12.78.62:39677), 20eae6ac52e34db5abb5c061cf4051ca (127.12.78.60:38949)
I20260809 20:24:24.838471 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.837004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:48058)
0809 20:24:24.837642 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.838042 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":477,"threads_started":1}
W20260809 20:24:24.838810 19919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
I20260809 20:24:24.839506 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.838434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:48066)
0809 20:24:24.839046 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.839216 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":810,"threads_started":1}
I20260809 20:24:24.841485 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.840613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:48074)
0809 20:24:24.840985 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.841230 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
W20260809 20:24:24.841771 19919 leader_election.cc:334] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ccd8c1717f7b4ce8b2489ede55f20522 (127.12.78.62:39677): Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
W20260809 20:24:24.842681 19718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
I20260809 20:24:24.846715 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.837207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:39915 (local address 127.0.0.1:40418)
0809 20:24:24.838744 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.838767 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:24.838955 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:24.839922 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:24.839930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:24.839958 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:24.840236 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:24.840255 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.842082 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.842090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:24.844101 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:24.844116 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.845893 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.845902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:24.846116 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:24.846162 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 20:24:24.846278 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 20:24:24.846436 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"thread_start_us":1885,"threads_started":1}
I20260809 20:24:24.847041 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.837335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.12.78.61:39915)
0809 20:24:24.838422 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:24.838428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:24.839317 (+ 889us) server_negotiation.cc:407] Connection header received
0809 20:24:24.839416 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:24.839424 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:24.839528 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:24.839636 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:24.840470 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.841884 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.844295 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.845667 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.845808 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:24.845928 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:24.846343 (+ 415us) server_negotiation.cc:299] Negotiation successful
0809 20:24:24.846593 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":966,"thread_start_us":805,"threads_started":1}
I20260809 20:24:24.847549 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.846898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:48092)
0809 20:24:24.847095 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.847316 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 20:24:24.847856 19718 leader_election.cc:334] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ccd8c1717f7b4ce8b2489ede55f20522 (127.12.78.62:39677): Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
I20260809 20:24:24.848647 19742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20eae6ac52e34db5abb5c061cf4051ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "45dcfa89acf6426c930dddd0e37f1254" is_pre_election: true
I20260809 20:24:24.848999 19742 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20eae6ac52e34db5abb5c061cf4051ca in term 1.
I20260809 20:24:24.849638 19917 leader_election.cc:302] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20eae6ac52e34db5abb5c061cf4051ca, 45dcfa89acf6426c930dddd0e37f1254; no voters: ccd8c1717f7b4ce8b2489ede55f20522
I20260809 20:24:24.850301 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.839584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38949 (local address 127.0.0.1:51912)
0809 20:24:24.839873 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.839893 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:24.840027 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:24.843408 (+ 3381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:24.843415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:24.843453 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:24.843805 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:24.843821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.845976 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.845982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:24.848045 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:24.848057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.849505 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.849511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:24.849702 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:24.849742 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 20:24:24.849853 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 20:24:24.850000 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":2771,"threads_started":1}
I20260809 20:24:24.850354 20509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 20:24:24.850473 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.840031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51912 (local address 127.12.78.60:38949)
0809 20:24:24.843011 (+ 2980us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:24.843019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:24.843046 (+ 27us) server_negotiation.cc:407] Connection header received
0809 20:24:24.843130 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:24.843137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:24.843225 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:24.843987 (+ 762us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:24.844312 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.845806 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.848206 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.849311 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.849455 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:24.849550 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:24.849893 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 20:24:24.850105 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2788,"thread_start_us":3729,"threads_started":1}
I20260809 20:24:24.850482 20509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 1 FOLLOWER]: Starting leader election (detected failure of leader ccd8c1717f7b4ce8b2489ede55f20522)
I20260809 20:24:24.850695 20509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 1 FOLLOWER]: Advancing to term 2
I20260809 20:24:24.851068 19944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45dcfa89acf6426c930dddd0e37f1254" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "20eae6ac52e34db5abb5c061cf4051ca" is_pre_election: true
I20260809 20:24:24.853044 20509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } }
I20260809 20:24:24.853523 19944 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45dcfa89acf6426c930dddd0e37f1254 in current term 2: Already voted for candidate 20eae6ac52e34db5abb5c061cf4051ca in this term.
I20260809 20:24:24.853787 20509 leader_election.cc:288] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [CANDIDATE]: Term 2 election: Requested vote from peers ccd8c1717f7b4ce8b2489ede55f20522 (127.12.78.62:39677), 45dcfa89acf6426c930dddd0e37f1254 (127.12.78.61:39915)
I20260809 20:24:24.854343 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.853876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:48104)
0809 20:24:24.854060 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.854156 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 20:24:24.854460 19742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20eae6ac52e34db5abb5c061cf4051ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "45dcfa89acf6426c930dddd0e37f1254"
I20260809 20:24:24.854712 19742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 1 FOLLOWER]: Advancing to term 2
I20260809 20:24:24.854934 19718 leader_election.cc:302] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45dcfa89acf6426c930dddd0e37f1254; no voters: 20eae6ac52e34db5abb5c061cf4051ca, ccd8c1717f7b4ce8b2489ede55f20522
I20260809 20:24:24.855414 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.855074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:48120)
0809 20:24:24.855227 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.855314 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 20:24:24.855592 19919 leader_election.cc:334] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ccd8c1717f7b4ce8b2489ede55f20522 (127.12.78.62:39677): Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
I20260809 20:24:24.857343 19742 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20eae6ac52e34db5abb5c061cf4051ca in term 2.
I20260809 20:24:24.857591 20510 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 20:24:24.857846 19917 leader_election.cc:302] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20eae6ac52e34db5abb5c061cf4051ca, 45dcfa89acf6426c930dddd0e37f1254; no voters: ccd8c1717f7b4ce8b2489ede55f20522
I20260809 20:24:24.858246 20509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 2 FOLLOWER]: Leader election won for term 2
I20260809 20:24:24.858692 20509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [term 2 LEADER]: Becoming Leader. State: Replica: 20eae6ac52e34db5abb5c061cf4051ca, State: Running, Role: LEADER
I20260809 20:24:24.859076 20509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } }
I20260809 20:24:24.861315 20519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20eae6ac52e34db5abb5c061cf4051ca. Latest consensus state: current_term: 2 leader_uuid: "20eae6ac52e34db5abb5c061cf4051ca" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } } }
I20260809 20:24:24.861763 20519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:24.862612 20522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:24.866674 20522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:24.867542 20522 catalog_manager.cc:1259] Loaded cluster ID: 958da1061cb0417788c3a84c366f3f25
I20260809 20:24:24.867611 20522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:24.868863 20522 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:24.869706 20522 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: Loaded TSK: 0
I20260809 20:24:24.870664 20522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:24:24.905416 19931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:24:24.912488 19931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:24:24.916631 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.916144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:48138)
0809 20:24:24.916316 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.916429 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 20:24:24.916791 19742 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [term 2 FOLLOWER]: Refusing update from remote peer 20eae6ac52e34db5abb5c061cf4051ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260809 20:24:24.917733 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.917307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:48142)
0809 20:24:24.917449 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.917580 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 20:24:24.917716 20519 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260809 20:24:24.918005 19919 consensus_peers.cc:487] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca -> Peer ccd8c1717f7b4ce8b2489ede55f20522 (127.12.78.62:39677): Couldn't send request to peer ccd8c1717f7b4ce8b2489ede55f20522. Status: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 20:24:24.920393 20510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20eae6ac52e34db5abb5c061cf4051ca. Latest consensus state: current_term: 2 leader_uuid: "20eae6ac52e34db5abb5c061cf4051ca" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } } }
I20260809 20:24:24.920723 20510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:24:24.922108 20509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "20eae6ac52e34db5abb5c061cf4051ca" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } } }
I20260809 20:24:24.922472 20509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:24.922961 20510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "20eae6ac52e34db5abb5c061cf4051ca" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccd8c1717f7b4ce8b2489ede55f20522" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39677 } } peers { permanent_uuid: "45dcfa89acf6426c930dddd0e37f1254" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 39915 } attrs { promote: false } } peers { permanent_uuid: "20eae6ac52e34db5abb5c061cf4051ca" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38949 } attrs { promote: false } } }
I20260809 20:24:24.923244 20510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45dcfa89acf6426c930dddd0e37f1254 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:24:24.942291 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.931311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:45283 (local address 127.0.0.1:35810)
0809 20:24:24.931475 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.931491 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:24.931659 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:24.933435 (+ 1776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:24.933444 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:24.933480 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:24.933913 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:24.933928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.935961 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.935970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:24.938878 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:24.938898 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.941134 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.941143 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:24.941420 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:24.941513 (+ 93us) client_negotiation.cc:769] Sending connection context
0809 20:24:24.941599 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 20:24:24.941793 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 20:24:24.943696 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.931516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.12.78.1:45283)
0809 20:24:24.932989 (+ 1473us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:24.933006 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:24.933033 (+ 27us) server_negotiation.cc:407] Connection header received
0809 20:24:24.933127 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:24.933135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:24.933241 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:24.934100 (+ 859us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:24.934312 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.935751 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.939161 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.940857 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.942927 (+ 2070us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:24.943099 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:24.943163 (+ 64us) server_negotiation.cc:299] Negotiation successful
0809 20:24:24.943349 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1355,"thread_start_us":1225,"threads_started":1}
I20260809 20:24:24.946066 20050 tablet_service.cc:1467] Processing CreateTablet for tablet 36cb24bc6fd64a538e1a880c0263d7cd (DEFAULT_TABLE table=test-table [id=8b4cd29d1a8d4719a1a3627810b6f718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:24.946579 20050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cb24bc6fd64a538e1a880c0263d7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:24.951517 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.933081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:43915 (local address 127.0.0.1:55226)
0809 20:24:24.942575 (+ 9494us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.942598 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:24.942721 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:24.944770 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:24.944779 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:24.944817 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:24.945091 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:24.945105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.946766 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.946772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:24.948903 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:24.948919 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.950693 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.950699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:24.950924 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:24.950976 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 20:24:24.951052 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 20:24:24.951195 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9377,"thread_start_us":11845,"threads_started":1}
I20260809 20:24:24.952661 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.933232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.12.78.3:43915)
0809 20:24:24.944287 (+ 11055us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:24.944298 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:24.944326 (+ 28us) server_negotiation.cc:407] Connection header received
0809 20:24:24.944423 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:24.944433 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:24.944536 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:24.945270 (+ 734us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:24.945486 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.946575 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.949142 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.950481 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.951749 (+ 1268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:24.951868 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:24.951923 (+ 55us) server_negotiation.cc:299] Negotiation successful
0809 20:24:24.952352 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10873,"thread_start_us":11372,"threads_started":1}
I20260809 20:24:24.955137 20532 tablet_bootstrap.cc:492] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812: Bootstrap starting.
I20260809 20:24:24.957039 20361 tablet_service.cc:1467] Processing CreateTablet for tablet 36cb24bc6fd64a538e1a880c0263d7cd (DEFAULT_TABLE table=test-table [id=8b4cd29d1a8d4719a1a3627810b6f718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:24.957194 20532 tablet_bootstrap.cc:654] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:24.957643 20361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cb24bc6fd64a538e1a880c0263d7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:24.958747 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.931773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:37183 (local address 127.0.0.1:50012)
0809 20:24:24.931956 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.931981 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:24.932133 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:24.953083 (+ 20950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:24.953088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:24.953115 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:24.953291 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:24.953300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.954830 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.954836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:24.956241 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:24.956256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.958014 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.958020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:24.958223 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:24.958255 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 20:24:24.958346 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 20:24:24.958487 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 20:24:24.959359 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.932102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.12.78.2:37183)
0809 20:24:24.952512 (+ 20410us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:24.952517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:24.952533 (+ 16us) server_negotiation.cc:407] Connection header received
0809 20:24:24.952649 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:24.952656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:24.952903 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:24.953444 (+ 541us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:24.953638 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.954640 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.956873 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:24.957862 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:24.958013 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:24.958111 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:24.958747 (+ 636us) server_negotiation.cc:299] Negotiation successful
0809 20:24:24.959056 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20292,"thread_start_us":20200,"threads_started":1}
I20260809 20:24:24.960129 20206 tablet_service.cc:1467] Processing CreateTablet for tablet 36cb24bc6fd64a538e1a880c0263d7cd (DEFAULT_TABLE table=test-table [id=8b4cd29d1a8d4719a1a3627810b6f718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:24.960623 20206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cb24bc6fd64a538e1a880c0263d7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:24.961309 20532 tablet_bootstrap.cc:492] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812: No bootstrap required, opened a new log
I20260809 20:24:24.961494 20532 ts_tablet_manager.cc:1397] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260809 20:24:24.962291 20532 raft_consensus.cc:348] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:24.962536 20532 raft_consensus.cc:374] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:24.962631 20532 raft_consensus.cc:729] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24068fc4286f45a4bfe48c531efac812, State: Initialized, Role: FOLLOWER
I20260809 20:24:24.962867 20532 consensus_queue.cc:260] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:24.963721 20532 ts_tablet_manager.cc:1428] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 20:24:24.969192 20537 tablet_bootstrap.cc:492] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607: Bootstrap starting.
I20260809 20:24:24.971216 20538 tablet_bootstrap.cc:492] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b: Bootstrap starting.
I20260809 20:24:24.971635 20537 tablet_bootstrap.cc:654] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:24.973254 20538 tablet_bootstrap.cc:654] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:24.973538 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.964406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.1:38091)
0809 20:24:24.973075 (+ 8669us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.973224 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8485,"thread_start_us":8364,"threads_started":1}
I20260809 20:24:24.976357 20121 heartbeater.cc:503] Master 127.12.78.60:38949 was elected leader, sending a full tablet report...
I20260809 20:24:24.976888 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.975714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.1:48011)
0809 20:24:24.976237 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.976360 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268}
I20260809 20:24:24.978459 20538 tablet_bootstrap.cc:492] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b: No bootstrap required, opened a new log
I20260809 20:24:24.978473 20537 tablet_bootstrap.cc:492] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607: No bootstrap required, opened a new log
I20260809 20:24:24.978659 20537 ts_tablet_manager.cc:1397] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260809 20:24:24.978816 20538 ts_tablet_manager.cc:1397] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 20:24:24.979530 20538 raft_consensus.cc:348] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:24.979774 20538 raft_consensus.cc:374] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:24.979879 20538 raft_consensus.cc:729] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc594aa5090a47c5b511a917b5c7b24b, State: Initialized, Role: FOLLOWER
I20260809 20:24:24.980000 20537 raft_consensus.cc:348] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:24.980139 20538 consensus_queue.cc:260] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:24.980335 20537 raft_consensus.cc:374] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:24.980456 20537 raft_consensus.cc:729] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4d517e110a40bb9f5f3caae6e3e607, State: Initialized, Role: FOLLOWER
I20260809 20:24:24.980788 20537 consensus_queue.cc:260] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:24.981066 20538 ts_tablet_manager.cc:1428] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 20:24:24.982885 20537 ts_tablet_manager.cc:1428] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260809 20:24:24.983594 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.981489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.2:45733)
0809 20:24:24.982151 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.983253 (+ 1102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"spinlock_wait_cycles":40192,"thread_start_us":440,"threads_started":1}
I20260809 20:24:24.984760 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.983807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.3:37789)
0809 20:24:24.984352 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.984471 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":287,"threads_started":1}
I20260809 20:24:24.984881 20278 heartbeater.cc:503] Master 127.12.78.60:38949 was elected leader, sending a full tablet report...
I20260809 20:24:24.986945 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.984780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.2:50589)
0809 20:24:24.985702 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.986319 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":803}
I20260809 20:24:24.987115 20429 heartbeater.cc:503] Master 127.12.78.60:38949 was elected leader, sending a full tablet report...
I20260809 20:24:24.987604 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:24.986712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.3:58677)
0809 20:24:24.987328 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:24.987422 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380}
I20260809 20:24:25.036633 20541 raft_consensus.cc:482] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:24:25.036883 20541 raft_consensus.cc:504] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:25.037710 20541 leader_election.cc:288] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24068fc4286f45a4bfe48c531efac812 (127.12.78.1:45283), 5d4d517e110a40bb9f5f3caae6e3e607 (127.12.78.3:43915)
I20260809 20:24:25.038312 20071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb24bc6fd64a538e1a880c0263d7cd" candidate_uuid: "fc594aa5090a47c5b511a917b5c7b24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24068fc4286f45a4bfe48c531efac812" is_pre_election: true
I20260809 20:24:25.038410 20381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb24bc6fd64a538e1a880c0263d7cd" candidate_uuid: "fc594aa5090a47c5b511a917b5c7b24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" is_pre_election: true
I20260809 20:24:25.038641 20071 raft_consensus.cc:2454] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc594aa5090a47c5b511a917b5c7b24b in term 0.
I20260809 20:24:25.038669 20381 raft_consensus.cc:2454] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc594aa5090a47c5b511a917b5c7b24b in term 0.
I20260809 20:24:25.039191 20158 leader_election.cc:302] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4d517e110a40bb9f5f3caae6e3e607, fc594aa5090a47c5b511a917b5c7b24b; no voters:
I20260809 20:24:25.039466 20541 raft_consensus.cc:2781] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:24:25.039595 20541 raft_consensus.cc:482] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:24:25.039709 20541 raft_consensus.cc:3037] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:25.042006 20541 raft_consensus.cc:504] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:25.042502 20541 leader_election.cc:288] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [CANDIDATE]: Term 1 election: Requested vote from peers 24068fc4286f45a4bfe48c531efac812 (127.12.78.1:45283), 5d4d517e110a40bb9f5f3caae6e3e607 (127.12.78.3:43915)
I20260809 20:24:25.042997 20071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb24bc6fd64a538e1a880c0263d7cd" candidate_uuid: "fc594aa5090a47c5b511a917b5c7b24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24068fc4286f45a4bfe48c531efac812"
I20260809 20:24:25.043074 20381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb24bc6fd64a538e1a880c0263d7cd" candidate_uuid: "fc594aa5090a47c5b511a917b5c7b24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4d517e110a40bb9f5f3caae6e3e607"
I20260809 20:24:25.043193 20071 raft_consensus.cc:3037] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:25.043339 20381 raft_consensus.cc:3037] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:25.045204 20071 raft_consensus.cc:2454] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc594aa5090a47c5b511a917b5c7b24b in term 1.
I20260809 20:24:25.045576 20381 raft_consensus.cc:2454] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc594aa5090a47c5b511a917b5c7b24b in term 1.
I20260809 20:24:25.045660 20158 leader_election.cc:302] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24068fc4286f45a4bfe48c531efac812, fc594aa5090a47c5b511a917b5c7b24b; no voters:
I20260809 20:24:25.045961 20541 raft_consensus.cc:2781] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:25.046466 20541 raft_consensus.cc:686] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [term 1 LEADER]: Becoming Leader. State: Replica: fc594aa5090a47c5b511a917b5c7b24b, State: Running, Role: LEADER
I20260809 20:24:25.046811 20541 consensus_queue.cc:237] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } }
I20260809 20:24:25.047681 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:25.047154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.2:55721)
0809 20:24:25.047362 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:25.047475 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 20:24:25.049700 19933 catalog_manager.cc:5495] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b reported cstate change: term changed from 0 to 1, leader changed from <none> to fc594aa5090a47c5b511a917b5c7b24b (127.12.78.2). New cstate: current_term: 1 leader_uuid: "fc594aa5090a47c5b511a917b5c7b24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc594aa5090a47c5b511a917b5c7b24b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 } health_report { overall_health: UNKNOWN } } }
I20260809 20:24:25.086709 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:25.082097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50016 (local address 127.12.78.2:37183)
0809 20:24:25.082290 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:25.082295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:25.082331 (+ 36us) server_negotiation.cc:407] Connection header received
0809 20:24:25.082490 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:25.082499 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:25.082566 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:25.082642 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:24:25.083014 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:25.083723 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:25.085044 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:25.085630 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:25.085734 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:25.086067 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:24:25.086171 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:24:25.086237 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:25.086364 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 20:24:25.086426 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 20:24:25.090502 20071 raft_consensus.cc:1261] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812 [term 1 FOLLOWER]: Refusing update from remote peer fc594aa5090a47c5b511a917b5c7b24b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:25.090516 20381 raft_consensus.cc:1261] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607 [term 1 FOLLOWER]: Refusing update from remote peer fc594aa5090a47c5b511a917b5c7b24b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:25.091256 20545 consensus_queue.cc:1035] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [LEADER]: Connected to new peer: Peer: permanent_uuid: "24068fc4286f45a4bfe48c531efac812" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:25.091732 20541 consensus_queue.cc:1035] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4d517e110a40bb9f5f3caae6e3e607" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 20:24:25.093537 20281 tablet.cc:2369] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:24:25.095165 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:25.094664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.3:57295)
0809 20:24:25.094833 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:25.094958 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260809 20:24:25.095317 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:25.093800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.1:52227)
0809 20:24:25.094999 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:25.095121 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1080}
I20260809 20:24:25.098652 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:25.097613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.3:52243)
0809 20:24:25.098406 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:25.098504 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":684}
I20260809 20:24:25.099192 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:25.097929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.2:36715)
0809 20:24:25.098938 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:25.099040 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":879}
I20260809 20:24:25.104337 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:25.103880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.2:38347)
0809 20:24:25.104033 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:25.104127 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 20:24:25.104449 20550 mvcc.cc:205] Tried to move back new op lower bound from 7316713738602266624 to 7316713738434453504. Current Snapshot: MvccSnapshot[applied={T|T < 7316713738602266624}]
I20260809 20:24:25.108664 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:25.105030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.12.78.1:42261)
0809 20:24:25.108306 (+ 3276us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:25.108442 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3117}
I20260809 20:24:25.110565 19931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:25.110855 19931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:25.116607 19931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20eae6ac52e34db5abb5c061cf4051ca: Sending DeleteTablet for 3 replicas of tablet 36cb24bc6fd64a538e1a880c0263d7cd
I20260809 20:24:25.117689 20206 tablet_service.cc:1514] Processing DeleteTablet for tablet 36cb24bc6fd64a538e1a880c0263d7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:25 UTC) from {username='oryx', principal='oryx/127.12.78.60@KRBTEST.COM'} at 127.0.0.1:50012
I20260809 20:24:25.117875 20050 tablet_service.cc:1514] Processing DeleteTablet for tablet 36cb24bc6fd64a538e1a880c0263d7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:25 UTC) from {username='oryx', principal='oryx/127.12.78.60@KRBTEST.COM'} at 127.0.0.1:35810
I20260809 20:24:25.118311 20361 tablet_service.cc:1514] Processing DeleteTablet for tablet 36cb24bc6fd64a538e1a880c0263d7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:25 UTC) from {username='oryx', principal='oryx/127.12.78.60@KRBTEST.COM'} at 127.0.0.1:55226
I20260809 20:24:25.121079 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 19907
I20260809 20:24:25.130376 20558 tablet_replica.cc:330] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812: stopping tablet replica
I20260809 20:24:25.130643 20557 tablet_replica.cc:330] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b: stopping tablet replica
I20260809 20:24:25.130592 20558 raft_consensus.cc:2229] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:25.130899 20558 raft_consensus.cc:2258] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:25.130926 20557 raft_consensus.cc:2229] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:24:25.131501 20557 raft_consensus.cc:2258] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:25.132329 20558 ts_tablet_manager.cc:1905] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:25.132592 20559 tablet_replica.cc:330] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607: stopping tablet replica
I20260809 20:24:25.132773 20557 ts_tablet_manager.cc:1905] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:25.132777 20559 raft_consensus.cc:2229] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:25.133054 20559 raft_consensus.cc:2258] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:25.134250 20559 ts_tablet_manager.cc:1905] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:25.139434 20557 ts_tablet_manager.cc:1918] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:25.139653 20557 log.cc:1196] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/36cb24bc6fd64a538e1a880c0263d7cd
I20260809 20:24:25.140192 20557 ts_tablet_manager.cc:1939] T 36cb24bc6fd64a538e1a880c0263d7cd P fc594aa5090a47c5b511a917b5c7b24b: Deleting consensus metadata
I20260809 20:24:25.141489 19917 catalog_manager.cc:4841] TS fc594aa5090a47c5b511a917b5c7b24b (127.12.78.2:37183): tablet 36cb24bc6fd64a538e1a880c0263d7cd (table test-table [id=8b4cd29d1a8d4719a1a3627810b6f718]) successfully deleted
I20260809 20:24:25.141743 20558 ts_tablet_manager.cc:1918] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:25.141912 20558 log.cc:1196] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/36cb24bc6fd64a538e1a880c0263d7cd
I20260809 20:24:25.142516 20558 ts_tablet_manager.cc:1939] T 36cb24bc6fd64a538e1a880c0263d7cd P 24068fc4286f45a4bfe48c531efac812: Deleting consensus metadata
I20260809 20:24:25.143859 19919 catalog_manager.cc:4841] TS 24068fc4286f45a4bfe48c531efac812 (127.12.78.1:45283): tablet 36cb24bc6fd64a538e1a880c0263d7cd (table test-table [id=8b4cd29d1a8d4719a1a3627810b6f718]) successfully deleted
I20260809 20:24:25.149391 20559 ts_tablet_manager.cc:1918] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:25.149593 20559 log.cc:1196] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/36cb24bc6fd64a538e1a880c0263d7cd
I20260809 20:24:25.150096 20559 ts_tablet_manager.cc:1939] T 36cb24bc6fd64a538e1a880c0263d7cd P 5d4d517e110a40bb9f5f3caae6e3e607: Deleting consensus metadata
I20260809 20:24:25.151234 19919 catalog_manager.cc:4841] TS 5d4d517e110a40bb9f5f3caae6e3e607 (127.12.78.3:43915): tablet 36cb24bc6fd64a538e1a880c0263d7cd (table test-table [id=8b4cd29d1a8d4719a1a3627810b6f718]) successfully deleted
I20260809 20:24:25.162429 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:25.123385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.12.78.1:45283)
0809 20:24:25.123664 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:25.123673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:25.149176 (+ 25503us) server_negotiation.cc:407] Connection header received
0809 20:24:25.149398 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:25.149403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:25.149490 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:25.149594 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:25.151260 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:25.152033 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:25.153370 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:25.153794 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:25.153934 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:25.156534 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:25.156609 (+ 75us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:25.156632 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:25.156700 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:25.159461 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:25.160173 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:25.160182 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:25.160185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:25.160351 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:25.160851 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:25.160858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:25.160861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:25.161251 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:25.161566 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:25.161878 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 20:24:25.162110 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 20:24:25.163118 20115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:25.256976 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 19907
I20260809 20:24:25.277563 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 20133
I20260809 20:24:25.292286 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:25.278311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.12.78.2:37183)
0809 20:24:25.278487 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:25.278491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:25.278737 (+ 246us) server_negotiation.cc:407] Connection header received
0809 20:24:25.278885 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:25.278889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:25.278952 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:25.279038 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:25.280338 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:25.281143 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:25.282364 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:25.282797 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:25.282894 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:25.285832 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:25.285849 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:25.285867 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:25.285928 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:25.289480 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:25.290082 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:25.290089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:25.290091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:25.290274 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:25.290633 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:25.290636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:25.290638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:25.291064 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:25.291284 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:25.291684 (+ 400us) server_negotiation.cc:299] Negotiation successful
0809 20:24:25.291897 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 20:24:25.293092 20272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:25.419116 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:25.418676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:48144)
0809 20:24:25.418854 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:25.418949 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 20:24:25.420644 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:25.420177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39677 (local address 127.0.0.1:48154)
0809 20:24:25.420334 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:25.420419 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 20:24:25.422856 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 20133
I20260809 20:24:25.448354 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 20291
I20260809 20:24:25.461920 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:25.449207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.12.78.3:43915)
0809 20:24:25.449406 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:25.449411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:25.449429 (+ 18us) server_negotiation.cc:407] Connection header received
0809 20:24:25.449527 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:25.449531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:25.449590 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:25.449691 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:25.451050 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:25.451768 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:25.453219 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:25.453643 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:25.453785 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:25.456283 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:25.456299 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:25.456316 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:25.456356 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:25.459040 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:25.459710 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:25.459717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:25.459719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:25.459847 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:25.460240 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:25.460245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:25.460248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:25.460767 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:25.461071 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:25.461376 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 20:24:25.461598 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 20:24:25.462595 20424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:25.558732 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 20291
I20260809 20:24:25.579087 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 19587
I20260809 20:24:25.580163 19763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:25.699797 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 19587
I20260809 20:24:25.725039 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 19695
I20260809 20:24:25.726205 19965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:25.848173 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 19695
2026-08-09T20:24:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10212 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:24:25 dist-test-slave-rnrw krb5kdc[20571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:24:25 dist-test-slave-rnrw krb5kdc[20571](info): set up 2 sockets
Aug 09 20:24:25 dist-test-slave-rnrw krb5kdc[20571](info): commencing operation
krb5kdc: starting...
W20260809 20:24:26.985718 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:24:27 dist-test-slave-rnrw krb5kdc[20571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:24:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:24:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.62.keytab.
Entry for principal oryx/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.62.keytab.
I20260809 20:24:27.226500 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44837
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.62.keytab
--principal=oryx/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:44837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:27.600883 20586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:27.601166 20586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:27.601316 20586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:27.608489 20586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:27.608666 20586 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 20:24:27.608747 20586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:27.608866 20586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:27.608961 20586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:27.609050 20586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:24:27.618878 20586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:44837
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.62.keytab
--principal=oryx/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:27.620110 20586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:27.621516 20586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:27.633191 20591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:27.633152 20594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:27.633251 20592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:27.634338 20586 server_base.cc:1034] running on GCE node
I20260809 20:24:27.635064 20586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:27.636433 20586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:27.637701 20586 hybrid_clock.cc:648] HybridClock initialized: now 1786307067637632 us; error 40 us; skew 500 ppm
I20260809 20:24:27.638038 20586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:27 dist-test-slave-rnrw krb5kdc[20571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307067, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:27.641134 20586 init.cc:377] Logged in from keytab as oryx/127.12.78.62@KRBTEST.COM (short username oryx)
I20260809 20:24:27.642580 20586 webserver.cc:460] Webserver started at http://127.12.78.62:45191/ using document root <none> and password file <none>
I20260809 20:24:27.643287 20586 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:27.643419 20586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:27.643798 20586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:27.647298 20586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "4578952b05f448d3863f7aedae942f49"
format_stamp: "Formatted at 2026-08-09 20:24:27 on dist-test-slave-rnrw"
server_key: "49632cb2067df997d54bac6aaddad3ea"
server_key_iv: "5b19697be21c4300d5e58dce1e774a41"
server_key_version: "encryptionkey@0"
I20260809 20:24:27.648110 20586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "4578952b05f448d3863f7aedae942f49"
format_stamp: "Formatted at 2026-08-09 20:24:27 on dist-test-slave-rnrw"
server_key: "49632cb2067df997d54bac6aaddad3ea"
server_key_iv: "5b19697be21c4300d5e58dce1e774a41"
server_key_version: "encryptionkey@0"
I20260809 20:24:27.653131 20586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 20:24:27.656466 20601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:27.657820 20586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:24:27.658002 20586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "4578952b05f448d3863f7aedae942f49"
format_stamp: "Formatted at 2026-08-09 20:24:27 on dist-test-slave-rnrw"
server_key: "49632cb2067df997d54bac6aaddad3ea"
server_key_iv: "5b19697be21c4300d5e58dce1e774a41"
server_key_version: "encryptionkey@0"
I20260809 20:24:27.658229 20586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:27.675748 20586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:27.679849 20586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:27.680130 20586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:27.702309 20586 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:44837
I20260809 20:24:27.702328 20664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:44837 every 8 connection(s)
I20260809 20:24:27.703651 20586 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:24:27.707336 20665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:27.711053 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 20586
I20260809 20:24:27.711272 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:24:27.712118 12600 external_mini_cluster.cc:1428] Setting key 634906982c57d3bdff61864087f0f9c0
I20260809 20:24:27.720383 20665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49: Bootstrap starting.
I20260809 20:24:27.723511 20665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:27.724488 20665 log.cc:824] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49: Log is configured to *not* fsync() on all Append() calls
Aug 09 20:24:27 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
I20260809 20:24:27.727566 20665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49: No bootstrap required, opened a new log
I20260809 20:24:27.735766 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:27.714402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.12.78.62:44837)
0809 20:24:27.715154 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:27.715165 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:27.715206 (+ 41us) server_negotiation.cc:407] Connection header received
0809 20:24:27.717011 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:27.717033 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:27.717430 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:27.717830 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:27.719345 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:27.720764 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:27.722017 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:27.722681 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:27.722873 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:27.727058 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:27.727069 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:27.727088 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:27.727139 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:27.731166 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:27.731844 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:27.731853 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:27.731866 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:27.732071 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:27.732438 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:27.732443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:27.732446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:27.733172 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:27.733479 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:27.733949 (+ 470us) server_negotiation.cc:299] Negotiation successful
0809 20:24:27.734437 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":345,"threads_started":1}
I20260809 20:24:27.738358 20665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4578952b05f448d3863f7aedae942f49" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44837 } }
I20260809 20:24:27.738818 20665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:27.738917 20665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4578952b05f448d3863f7aedae942f49, State: Initialized, Role: FOLLOWER
I20260809 20:24:27.739531 20665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4578952b05f448d3863f7aedae942f49" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44837 } }
I20260809 20:24:27.739749 20665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:24:27.739904 20665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:24:27.740069 20665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:27.742266 20665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4578952b05f448d3863f7aedae942f49" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44837 } }
I20260809 20:24:27.742763 20665 leader_election.cc:302] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4578952b05f448d3863f7aedae942f49; no voters:
I20260809 20:24:27.743341 20665 leader_election.cc:288] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:24:27.743496 20670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:27.744679 20670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [term 1 LEADER]: Becoming Leader. State: Replica: 4578952b05f448d3863f7aedae942f49, State: Running, Role: LEADER
I20260809 20:24:27.745267 20670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4578952b05f448d3863f7aedae942f49" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44837 } }
I20260809 20:24:27.745594 20665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:27.748399 20671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4578952b05f448d3863f7aedae942f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4578952b05f448d3863f7aedae942f49" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44837 } } }
I20260809 20:24:27.749171 20671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:27.749167 20672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4578952b05f448d3863f7aedae942f49. Latest consensus state: current_term: 1 leader_uuid: "4578952b05f448d3863f7aedae942f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4578952b05f448d3863f7aedae942f49" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44837 } } }
I20260809 20:24:27.749836 20672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:27.751227 20680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:27.755239 20680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:27.763998 20680 catalog_manager.cc:1347] Generated new cluster ID: 20e936cb16f24d96a4297353de54c628
I20260809 20:24:27.764104 20680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:27.775780 20680 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:24:27.777158 20680 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:27.782485 20680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49: Generated new TSK 0
I20260809 20:24:27.783324 20680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab.
Entry for principal oryx/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab.
I20260809 20:24:27.849860 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44837
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab
--principal=oryx/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:28.190114 20692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:28.190383 20692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:28.190550 20692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:28.197858 20692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:28.197964 20692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:28.198143 20692 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 20:24:28.198269 20692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:24:28.207892 20692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab
--principal=oryx/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.78.62:44837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:28.209204 20692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:28.210569 20692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:28.223688 20697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:28.223593 20700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:28.223742 20698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:28.224843 20692 server_base.cc:1034] running on GCE node
I20260809 20:24:28.225534 20692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:28.226576 20692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:28.227840 20692 hybrid_clock.cc:648] HybridClock initialized: now 1786307068227770 us; error 31 us; skew 500 ppm
I20260809 20:24:28.228188 20692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:28 dist-test-slave-rnrw krb5kdc[20571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307068, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:28.231598 20692 init.cc:377] Logged in from keytab as oryx/127.12.78.1@KRBTEST.COM (short username oryx)
I20260809 20:24:28.233088 20692 webserver.cc:460] Webserver started at http://127.12.78.1:33055/ using document root <none> and password file <none>
I20260809 20:24:28.233773 20692 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:28.233906 20692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:28.234248 20692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:28.237360 20692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "e681dee7227443c5a6916bf22a8bcd46"
format_stamp: "Formatted at 2026-08-09 20:24:28 on dist-test-slave-rnrw"
server_key: "a9dd5d87bb918114bf9c0939072a6479"
server_key_iv: "3a42ab4b8cb3f2b7caf326c8933690f7"
server_key_version: "encryptionkey@0"
I20260809 20:24:28.238143 20692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "e681dee7227443c5a6916bf22a8bcd46"
format_stamp: "Formatted at 2026-08-09 20:24:28 on dist-test-slave-rnrw"
server_key: "a9dd5d87bb918114bf9c0939072a6479"
server_key_iv: "3a42ab4b8cb3f2b7caf326c8933690f7"
server_key_version: "encryptionkey@0"
I20260809 20:24:28.243597 20692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:24:28.247262 20709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:28.248514 20692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 20:24:28.248739 20692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "e681dee7227443c5a6916bf22a8bcd46"
format_stamp: "Formatted at 2026-08-09 20:24:28 on dist-test-slave-rnrw"
server_key: "a9dd5d87bb918114bf9c0939072a6479"
server_key_iv: "3a42ab4b8cb3f2b7caf326c8933690f7"
server_key_version: "encryptionkey@0"
I20260809 20:24:28.248960 20692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:28.269241 20692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:28.273697 20692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:28.273983 20692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:28.278301 20692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:28.278410 20692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:28.278654 20692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:28.278755 20692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 20:24:28 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307068, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
I20260809 20:24:28.301950 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:28.278766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:52301 (local address 127.12.78.62:44837)
0809 20:24:28.279315 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:28.279321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:28.280571 (+ 1250us) server_negotiation.cc:407] Connection header received
0809 20:24:28.282991 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:28.282997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:28.283099 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:28.283231 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:28.285454 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.286530 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.289324 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.289741 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.289880 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:28.294114 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:28.294122 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:28.294127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:28.294176 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:28.297698 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:28.298851 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:28.298855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:28.298857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:28.298980 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:28.299603 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:28.299607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:28.299609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:28.299873 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:28.299985 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:28.301192 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0809 20:24:28.301459 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":245,"threads_started":1}
I20260809 20:24:28.303917 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:28.279426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44837 (local address 127.12.78.1:52301)
0809 20:24:28.280357 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:28.280418 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:28.282689 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:28.283470 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:28.283481 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:28.284332 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:28.285259 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:28.285279 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.286693 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.286703 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:28.289167 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:28.289181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.290080 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.290090 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:28.290282 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:28.291037 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:28.291065 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:28.293760 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:28.297851 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:28.297864 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:28.297912 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:28.298703 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:28.299098 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:28.299106 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:28.299112 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:28.299448 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:28.300005 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:28.300017 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:28.300648 (+ 631us) client_negotiation.cc:769] Sending connection context
0809 20:24:28.301051 (+ 403us) client_negotiation.cc:240] Negotiation successful
0809 20:24:28.301803 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":412,"threads_started":1}
I20260809 20:24:28.327095 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:28.310845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44837 (local address 127.12.78.1:41443)
0809 20:24:28.311881 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:28.311908 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:28.312078 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:28.312444 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:28.312469 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:28.313347 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:28.313895 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:28.313913 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.315199 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.315205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:28.316247 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:28.316258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.317121 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.317131 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:28.317408 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:28.318127 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:28.318160 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:28.319546 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:28.323736 (+ 4190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:28.323744 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:28.323748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:28.324332 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:28.324768 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:28.324775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:28.324784 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:28.324955 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:28.325802 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:28.325809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:28.326043 (+ 234us) client_negotiation.cc:769] Sending connection context
0809 20:24:28.326229 (+ 186us) client_negotiation.cc:240] Negotiation successful
0809 20:24:28.326608 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":476,"threads_started":1}
I20260809 20:24:28.327750 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:28.311861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:41443 (local address 127.12.78.62:44837)
0809 20:24:28.312030 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:28.312034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:28.312051 (+ 17us) server_negotiation.cc:407] Connection header received
0809 20:24:28.312208 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:28.312211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:28.312273 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:28.312366 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:28.314097 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.315051 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.316392 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.316865 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.317039 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:28.319750 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:28.319755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:28.319758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:28.319790 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:28.323540 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:28.324456 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:28.324460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:28.324462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:28.324653 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:28.325088 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:28.325095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:28.325100 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:28.325525 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:28.325712 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:28.326686 (+ 974us) server_negotiation.cc:299] Negotiation successful
0809 20:24:28.327019 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 20:24:28.335986 20692 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:41219
I20260809 20:24:28.336015 20836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:41219 every 8 connection(s)
I20260809 20:24:28.337296 20692 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:24:28.338061 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 20692
I20260809 20:24:28.338315 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:24:28.339182 12600 external_mini_cluster.cc:1428] Setting key 83f777ad91bbab3e95b623132d004e53
I20260809 20:24:28.339639 20837 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44837
I20260809 20:24:28.339869 20837 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:28.340654 20837 heartbeater.cc:511] Master 127.12.78.62:44837 requested a full tablet report, sending...
I20260809 20:24:28.342751 20619 ts_manager.cc:194] Registered new tserver with Master: e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1:41219)
I20260809 20:24:28.344266 20619 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:52301
WARNING: no policy specified for oryx/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab.
Entry for principal oryx/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab.
I20260809 20:24:28.392238 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44837
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab
--principal=oryx/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:28.734313 20843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:28.734594 20843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:28.734692 20843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:28.741853 20843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:28.741954 20843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:28.742035 20843 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 20:24:28.742115 20843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:24:28.751725 20843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab
--principal=oryx/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.78.62:44837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:28.753051 20843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:28.754395 20843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:28.767700 20851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:28.767740 20848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:28.767786 20849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:28.773720 20843 server_base.cc:1034] running on GCE node
I20260809 20:24:28.774410 20843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:28.775290 20843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:28.776517 20843 hybrid_clock.cc:648] HybridClock initialized: now 1786307068776465 us; error 30 us; skew 500 ppm
I20260809 20:24:28.776872 20843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:28 dist-test-slave-rnrw krb5kdc[20571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307068, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:28.780038 20843 init.cc:377] Logged in from keytab as oryx/127.12.78.2@KRBTEST.COM (short username oryx)
I20260809 20:24:28.782786 20843 webserver.cc:460] Webserver started at http://127.12.78.2:39195/ using document root <none> and password file <none>
I20260809 20:24:28.783710 20843 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:28.783841 20843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:28.784240 20843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:28.787201 20843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "ba56fdec4011408c8d8bc08e7d4e33e1"
format_stamp: "Formatted at 2026-08-09 20:24:28 on dist-test-slave-rnrw"
server_key: "047cd2aad72d77adc72fdc2c63708748"
server_key_iv: "e8be6b8d1a56e4e64875c89c0df3789d"
server_key_version: "encryptionkey@0"
I20260809 20:24:28.787992 20843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "ba56fdec4011408c8d8bc08e7d4e33e1"
format_stamp: "Formatted at 2026-08-09 20:24:28 on dist-test-slave-rnrw"
server_key: "047cd2aad72d77adc72fdc2c63708748"
server_key_iv: "e8be6b8d1a56e4e64875c89c0df3789d"
server_key_version: "encryptionkey@0"
I20260809 20:24:28.793068 20843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 20:24:28.796380 20858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:28.797773 20843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:28.797935 20843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "ba56fdec4011408c8d8bc08e7d4e33e1"
format_stamp: "Formatted at 2026-08-09 20:24:28 on dist-test-slave-rnrw"
server_key: "047cd2aad72d77adc72fdc2c63708748"
server_key_iv: "e8be6b8d1a56e4e64875c89c0df3789d"
server_key_version: "encryptionkey@0"
I20260809 20:24:28.798172 20843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:28.835394 20843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:28.839960 20843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:28.840197 20843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:28.842845 20843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:28.842990 20843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:28.843168 20843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:28.843245 20843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 20:24:28 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307068, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
I20260809 20:24:28.868491 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:28.844424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44837 (local address 127.12.78.2:36081)
0809 20:24:28.845358 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:28.845407 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:28.847533 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:28.848533 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:28.848588 (+ 55us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:28.849511 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:28.850496 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:28.850519 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.852120 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.852127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:28.853959 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:28.853972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.855199 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.855212 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:28.855479 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:28.856211 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:28.856260 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:28.859192 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:28.863738 (+ 4546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:28.863751 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:28.863767 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:28.864232 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:28.864683 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:28.864689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:28.864696 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:28.864968 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:28.865744 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:28.865756 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:28.866206 (+ 450us) client_negotiation.cc:769] Sending connection context
0809 20:24:28.866606 (+ 400us) client_negotiation.cc:240] Negotiation successful
0809 20:24:28.867253 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":469,"threads_started":1}
I20260809 20:24:28.869701 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:28.843692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:36081 (local address 127.12.78.62:44837)
0809 20:24:28.844421 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:28.844429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:28.845537 (+ 1108us) server_negotiation.cc:407] Connection header received
0809 20:24:28.847901 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:28.847907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:28.848014 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:28.848157 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:28.850744 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.851850 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.854112 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.854920 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.855085 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:28.859458 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:28.859467 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:28.859474 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:28.859530 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:28.863585 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:28.864391 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:28.864396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:28.864401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:28.864530 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:28.865128 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:28.865135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:28.865141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:28.865552 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:28.865760 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:28.869270 (+ 3510us) server_negotiation.cc:299] Negotiation successful
0809 20:24:28.869448 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":468,"threads_started":1}
I20260809 20:24:28.895740 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:28.879432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44837 (local address 127.12.78.2:42195)
0809 20:24:28.880234 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:28.880262 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:28.880445 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:28.880917 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:28.880921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:28.881766 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:28.882267 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:28.882284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.883790 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.883795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:28.885575 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:28.885592 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.886532 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.886538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:28.886740 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:28.887408 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:28.887449 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:28.888936 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:28.892907 (+ 3971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:28.892917 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:28.892922 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:28.893480 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:28.893956 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:28.893959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:28.893962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:28.894079 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:28.894674 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:28.894679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:28.894937 (+ 258us) client_negotiation.cc:769] Sending connection context
0809 20:24:28.895141 (+ 204us) client_negotiation.cc:240] Negotiation successful
0809 20:24:28.895405 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":467,"threads_started":1}
I20260809 20:24:28.896443 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:28.879342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:42195 (local address 127.12.78.62:44837)
0809 20:24:28.879531 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:28.879537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:28.880376 (+ 839us) server_negotiation.cc:407] Connection header received
0809 20:24:28.880695 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:28.880701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:28.880776 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:28.880884 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:28.882502 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.883636 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.885770 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:28.886373 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:28.886470 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:28.889226 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:28.889234 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:28.889241 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:28.889287 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:28.892768 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:28.893625 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:28.893634 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:28.893639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:28.893807 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:28.894202 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:28.894207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:28.894210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:28.894538 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:28.894702 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:28.896078 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0809 20:24:28.896245 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 20:24:28.901324 20843 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:37333
I20260809 20:24:28.901373 20984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:37333 every 8 connection(s)
I20260809 20:24:28.902674 20843 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:24:28.904661 20985 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44837
I20260809 20:24:28.904897 20985 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:28.905339 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 20843
I20260809 20:24:28.905506 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:24:28.905633 20985 heartbeater.cc:511] Master 127.12.78.62:44837 requested a full tablet report, sending...
I20260809 20:24:28.906023 12600 external_mini_cluster.cc:1428] Setting key 2e56f880fd075d87ed05f606495aad62
I20260809 20:24:28.907140 20619 ts_manager.cc:194] Registered new tserver with Master: ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2:37333)
I20260809 20:24:28.908219 20619 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:36081
WARNING: no policy specified for oryx/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab.
Entry for principal oryx/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab.
I20260809 20:24:28.959576 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44837
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab
--principal=oryx/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:29.292107 20989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:29.292402 20989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:29.292502 20989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:29.299906 20989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:29.300026 20989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:29.300119 20989 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 20:24:29.300204 20989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:24:29.310436 20989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab
--principal=oryx/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.78.62:44837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:29.311944 20989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:29.313427 20989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:29.326805 20994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:29.326831 20995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:29.326754 20997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:29.327899 20989 server_base.cc:1034] running on GCE node
I20260809 20:24:29.328509 20989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:29.329624 20989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:29.330865 20989 hybrid_clock.cc:648] HybridClock initialized: now 1786307069330762 us; error 70 us; skew 500 ppm
I20260809 20:24:29.331260 20989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:29 dist-test-slave-rnrw krb5kdc[20571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307069, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:29.334764 20989 init.cc:377] Logged in from keytab as oryx/127.12.78.3@KRBTEST.COM (short username oryx)
I20260809 20:24:29.336324 20989 webserver.cc:460] Webserver started at http://127.12.78.3:38555/ using document root <none> and password file <none>
I20260809 20:24:29.337188 20989 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:29.337282 20989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:29.337622 20989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:29.340646 20989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "170ecc40857b446f8034b62956e7f13b"
format_stamp: "Formatted at 2026-08-09 20:24:29 on dist-test-slave-rnrw"
server_key: "ca5f060f29c843442deda8f34ffcd86d"
server_key_iv: "560df72f87ca570ea632fa7978227412"
server_key_version: "encryptionkey@0"
I20260809 20:24:29.341512 20989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "170ecc40857b446f8034b62956e7f13b"
format_stamp: "Formatted at 2026-08-09 20:24:29 on dist-test-slave-rnrw"
server_key: "ca5f060f29c843442deda8f34ffcd86d"
server_key_iv: "560df72f87ca570ea632fa7978227412"
server_key_version: "encryptionkey@0"
I20260809 20:24:29.346689 20989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:24:29.347788 20837 heartbeater.cc:503] Master 127.12.78.62:44837 was elected leader, sending a full tablet report...
I20260809 20:24:29.350464 21004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:29.351843 20989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 20:24:29.352016 20989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "170ecc40857b446f8034b62956e7f13b"
format_stamp: "Formatted at 2026-08-09 20:24:29 on dist-test-slave-rnrw"
server_key: "ca5f060f29c843442deda8f34ffcd86d"
server_key_iv: "560df72f87ca570ea632fa7978227412"
server_key_version: "encryptionkey@0"
I20260809 20:24:29.352246 20989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:29.364482 20989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:29.368822 20989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:29.369091 20989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:29.372007 20989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:29.372138 20989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:29.372295 20989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:29.372368 20989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 20:24:29 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307069, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
I20260809 20:24:29.396760 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.373111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44837 (local address 127.12.78.3:33239)
0809 20:24:29.373905 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:29.373948 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:29.375756 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:29.376821 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:29.376829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:29.377583 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:29.378435 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.378483 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.380213 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.380221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.382572 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.382590 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.383441 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.383451 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.383692 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.384451 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:29.384472 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:29.387712 (+ 3240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:29.391697 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:29.391706 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:29.391722 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:29.392246 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:29.392704 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:29.392709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:29.392713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:29.392991 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:29.393766 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:29.393778 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:29.394224 (+ 446us) client_negotiation.cc:769] Sending connection context
0809 20:24:29.394636 (+ 412us) client_negotiation.cc:240] Negotiation successful
0809 20:24:29.395204 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":356,"threads_started":1}
I20260809 20:24:29.397351 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.372486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:33239 (local address 127.12.78.62:44837)
0809 20:24:29.372693 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:29.372702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:29.374063 (+ 1361us) server_negotiation.cc:407] Connection header received
0809 20:24:29.376169 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:29.376187 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:29.376339 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:29.376513 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:29.378679 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.380040 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.382756 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.383199 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.383346 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.387954 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:29.387959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:29.387962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:29.387996 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:29.391555 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:29.392373 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:29.392385 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:29.392388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:29.392527 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:29.393159 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:29.393164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:29.393167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:29.393605 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:29.393804 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:29.396790 (+ 2986us) server_negotiation.cc:299] Negotiation successful
0809 20:24:29.397028 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 20:24:29.418644 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.403043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44837 (local address 127.12.78.3:39849)
0809 20:24:29.403832 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:29.403853 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:29.404041 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:29.404613 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:29.404622 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:29.405328 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:29.405866 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.405889 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.407636 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.407641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.409523 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.409541 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.410575 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.410581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.410750 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.411386 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:29.411403 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:29.412156 (+ 753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:29.415536 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:29.415544 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:29.415552 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:29.416169 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:29.416664 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:29.416668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:29.416670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:29.416792 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:29.417443 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:29.417447 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:29.417643 (+ 196us) client_negotiation.cc:769] Sending connection context
0809 20:24:29.417852 (+ 209us) client_negotiation.cc:240] Negotiation successful
0809 20:24:29.418302 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":463,"threads_started":1}
I20260809 20:24:29.418788 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.403044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:39849 (local address 127.12.78.62:44837)
0809 20:24:29.403256 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:29.403264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:29.403964 (+ 700us) server_negotiation.cc:407] Connection header received
0809 20:24:29.404185 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:29.404191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:29.404290 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:29.404391 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:29.406034 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.407416 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.409740 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.410375 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.410524 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.412345 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:29.412394 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:29.412401 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:29.412443 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:29.415397 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:29.416325 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:29.416333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:29.416336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:29.416515 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:29.416914 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:29.416921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:29.416925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:29.417319 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:29.417485 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:29.417992 (+ 507us) server_negotiation.cc:299] Negotiation successful
0809 20:24:29.418271 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 20:24:29.426538 20989 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:41445
I20260809 20:24:29.426568 21128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:41445 every 8 connection(s)
I20260809 20:24:29.427904 20989 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:24:29.430202 21129 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44837
I20260809 20:24:29.430444 21129 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:29.431222 21129 heartbeater.cc:511] Master 127.12.78.62:44837 requested a full tablet report, sending...
I20260809 20:24:29.432687 20619 ts_manager.cc:194] Registered new tserver with Master: 170ecc40857b446f8034b62956e7f13b (127.12.78.3:41445)
I20260809 20:24:29.433183 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 20989
I20260809 20:24:29.433362 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:24:29.433925 12600 external_mini_cluster.cc:1428] Setting key e0752c2503e2696e07c782d965d6f247
I20260809 20:24:29.433923 20619 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:33239
I20260809 20:24:29.437078 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 20:24:29 dist-test-slave-rnrw krb5kdc[20571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307069, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 20:24:29 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:29 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 20:24:29.494297 20868 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:29.483657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.12.78.62:44837)
0809 20:24:29.483837 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:29.483841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:29.484179 (+ 338us) server_negotiation.cc:407] Connection header received
0809 20:24:29.484326 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:29.484331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:29.484397 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:29.484495 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:29.485927 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.486709 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.487982 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.488390 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.488491 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.494077 (+ 5586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51178: BlockingRecv error: recv got EOF from 127.0.0.1:51178 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 09 20:24:29 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307069, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
I20260809 20:24:29.511389 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.497591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51192 (local address 127.12.78.62:44837)
0809 20:24:29.497840 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:29.497849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:29.498145 (+ 296us) server_negotiation.cc:407] Connection header received
0809 20:24:29.498338 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:29.498346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:29.498415 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:29.498523 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:29.499748 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.500499 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.501802 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.502253 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.502402 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.506205 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:29.506210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:29.506213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:29.506248 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:29.509090 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:29.509720 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:29.509723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:29.509726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:29.509843 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:29.510153 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:29.510155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:29.510157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:29.510418 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:29.510628 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:29.510854 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 20:24:29.511019 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260809 20:24:29.515292 20619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:24:29.522306 20619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:24:29.565418 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.534573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42986 (local address 127.12.78.2:37333)
0809 20:24:29.555149 (+ 20576us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:29.555163 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:29.555828 (+ 665us) server_negotiation.cc:407] Connection header received
0809 20:24:29.556177 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:29.556189 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:29.556489 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:29.556715 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:29.558292 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.559839 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.562525 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.564285 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.564442 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.564697 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:29.564944 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 20:24:29.565119 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18661,"thread_start_us":20243,"threads_started":1}
I20260809 20:24:29.565784 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.533946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:37333 (local address 127.0.0.1:42986)
0809 20:24:29.555700 (+ 21754us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:29.555725 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:29.555951 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:29.557397 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:29.557404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:29.557446 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:29.558038 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.558052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.559989 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.559994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.562326 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.562343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.564475 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.564481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.564696 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.564761 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 20:24:29.564890 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 20:24:29.565087 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21582,"mutex_wait_us":162,"spinlock_wait_cycles":4480,"thread_start_us":11888,"threads_started":1}
I20260809 20:24:29.566395 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.533749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:41445 (local address 127.0.0.1:43366)
0809 20:24:29.546322 (+ 12573us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:29.547047 (+ 725us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:29.547267 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:29.557937 (+ 10670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:29.557941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:29.557970 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:29.558750 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.558762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.560429 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.560437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.563323 (+ 2886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.563332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.565310 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.565317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.565768 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.565809 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 20:24:29.565879 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 20:24:29.566022 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12240,"mutex_wait_us":90,"thread_start_us":21619,"threads_started":1}
I20260809 20:24:29.567414 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.534361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:41219 (local address 127.0.0.1:33392)
0809 20:24:29.556495 (+ 22134us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:29.556521 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:29.556678 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:29.557404 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:29.557410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:29.557458 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:29.558731 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.558747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.560525 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.560536 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.563317 (+ 2781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.563332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.565715 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.565723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.565890 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.565917 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 20:24:29.565989 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 20:24:29.567035 (+ 1046us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22009,"mutex_wait_us":181,"spinlock_wait_cycles":2944,"thread_start_us":21868,"threads_started":1}
I20260809 20:24:29.567572 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.534091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.12.78.3:41445)
0809 20:24:29.557351 (+ 23260us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:29.557360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:29.557394 (+ 34us) server_negotiation.cc:407] Connection header received
0809 20:24:29.557484 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:29.557492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:29.557721 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:29.557929 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:29.559003 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.560272 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.563557 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.565058 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.566730 (+ 1672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.566960 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:29.567030 (+ 70us) server_negotiation.cc:299] Negotiation successful
0809 20:24:29.567214 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23095,"thread_start_us":22959,"threads_started":1}
I20260809 20:24:29.568213 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.534998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.12.78.1:41219)
0809 20:24:29.556044 (+ 21046us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:29.556054 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:29.556678 (+ 624us) server_negotiation.cc:407] Connection header received
0809 20:24:29.556822 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:29.556833 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:29.557182 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:29.557381 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:29.558977 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.560311 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.563528 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.565058 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.565203 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.565417 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:29.567750 (+ 2333us) server_negotiation.cc:299] Negotiation successful
0809 20:24:29.567931 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20875,"thread_start_us":20668,"threads_started":1}
I20260809 20:24:29.569741 20912 tablet_service.cc:1467] Processing CreateTablet for tablet 706dd42119594228adaf900ea3f24afe (DEFAULT_TABLE table=test-table [id=f5fd53974b6d46678b2e329fe5b7f099]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:29.571266 20912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706dd42119594228adaf900ea3f24afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:29.570914 20764 tablet_service.cc:1467] Processing CreateTablet for tablet 706dd42119594228adaf900ea3f24afe (DEFAULT_TABLE table=test-table [id=f5fd53974b6d46678b2e329fe5b7f099]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:29.571038 21057 tablet_service.cc:1467] Processing CreateTablet for tablet 706dd42119594228adaf900ea3f24afe (DEFAULT_TABLE table=test-table [id=f5fd53974b6d46678b2e329fe5b7f099]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:29.572134 20764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706dd42119594228adaf900ea3f24afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:29.572382 21057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706dd42119594228adaf900ea3f24afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:29.582548 21154 tablet_bootstrap.cc:492] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46: Bootstrap starting.
I20260809 20:24:29.586421 21154 tablet_bootstrap.cc:654] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:29.586401 21156 tablet_bootstrap.cc:492] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1: Bootstrap starting.
I20260809 20:24:29.587728 21154 log.cc:824] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:29.589264 21157 tablet_bootstrap.cc:492] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b: Bootstrap starting.
I20260809 20:24:29.590328 21156 tablet_bootstrap.cc:654] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:29.590816 21154 tablet_bootstrap.cc:492] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46: No bootstrap required, opened a new log
I20260809 20:24:29.591014 21154 ts_tablet_manager.cc:1397] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260809 20:24:29.591588 21156 log.cc:824] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:29.592875 21157 tablet_bootstrap.cc:654] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:29.594157 21157 log.cc:824] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:29.595158 21156 tablet_bootstrap.cc:492] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1: No bootstrap required, opened a new log
I20260809 20:24:29.595494 21156 ts_tablet_manager.cc:1397] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260809 20:24:29.597234 21157 tablet_bootstrap.cc:492] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b: No bootstrap required, opened a new log
I20260809 20:24:29.597481 21157 ts_tablet_manager.cc:1397] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 20:24:29.599488 21154 raft_consensus.cc:348] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } }
I20260809 20:24:29.599888 21154 raft_consensus.cc:374] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:29.599988 21154 raft_consensus.cc:729] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e681dee7227443c5a6916bf22a8bcd46, State: Initialized, Role: FOLLOWER
I20260809 20:24:29.600656 21154 consensus_queue.cc:260] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } }
I20260809 20:24:29.603482 21154 ts_tablet_manager.cc:1428] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260809 20:24:29.607142 21156 raft_consensus.cc:348] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } }
I20260809 20:24:29.607606 21156 raft_consensus.cc:374] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:29.607717 21156 raft_consensus.cc:729] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba56fdec4011408c8d8bc08e7d4e33e1, State: Initialized, Role: FOLLOWER
I20260809 20:24:29.608335 21156 consensus_queue.cc:260] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } }
I20260809 20:24:29.608353 21157 raft_consensus.cc:348] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } }
I20260809 20:24:29.608773 21157 raft_consensus.cc:374] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:29.608894 21157 raft_consensus.cc:729] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170ecc40857b446f8034b62956e7f13b, State: Initialized, Role: FOLLOWER
I20260809 20:24:29.609534 20985 heartbeater.cc:503] Master 127.12.78.62:44837 was elected leader, sending a full tablet report...
I20260809 20:24:29.609494 21157 consensus_queue.cc:260] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } }
I20260809 20:24:29.610250 21156 ts_tablet_manager.cc:1428] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260809 20:24:29.610632 21129 heartbeater.cc:503] Master 127.12.78.62:44837 was elected leader, sending a full tablet report...
I20260809 20:24:29.611143 21157 ts_tablet_manager.cc:1428] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260809 20:24:29.654685 20986 tablet.cc:2369] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:24:29.679772 21130 tablet.cc:2369] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:24:29.748693 21163 raft_consensus.cc:482] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:24:29.749061 21163 raft_consensus.cc:504] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } }
I20260809 20:24:29.750980 21163 leader_election.cc:288] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2:37333), e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1:41219)
I20260809 20:24:29.758163 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.751357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:33793 (local address 127.12.78.2:37333)
0809 20:24:29.751601 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:29.751609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:29.751741 (+ 132us) server_negotiation.cc:407] Connection header received
0809 20:24:29.751963 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:29.751971 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:29.752074 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:29.752190 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:29.752700 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.753791 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.756127 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.757318 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.757519 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.757634 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:29.757777 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 20:24:29.757921 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 20:24:29.758158 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.751400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:37333 (local address 127.12.78.3:33793)
0809 20:24:29.751607 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:29.751656 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:29.751793 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:29.752239 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:29.752244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:29.752282 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:29.752515 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.752527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.753929 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.753933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.755974 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.755989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.757468 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.757472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.757608 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.757645 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 20:24:29.757723 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 20:24:29.757815 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 20:24:29.758744 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.751521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:50065 (local address 127.12.78.1:41219)
0809 20:24:29.751780 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:29.751786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:29.752531 (+ 745us) server_negotiation.cc:407] Connection header received
0809 20:24:29.752708 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:29.752712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:29.752799 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:29.752898 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:29.753361 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.754260 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.756496 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.757793 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.757921 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.758026 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:29.758429 (+ 403us) server_negotiation.cc:299] Negotiation successful
0809 20:24:29.758561 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260809 20:24:29.759289 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.751773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:41219 (local address 127.12.78.3:50065)
0809 20:24:29.752427 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:29.752448 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:29.752595 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:29.752931 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:29.752934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:29.752954 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:29.753217 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.753224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.754391 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.754396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.756328 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:29.756338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.758032 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.758038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:29.758224 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.758259 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 20:24:29.758934 (+ 675us) client_negotiation.cc:240] Negotiation successful
0809 20:24:29.759038 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":303,"threads_started":1}
I20260809 20:24:29.759256 20938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "706dd42119594228adaf900ea3f24afe" candidate_uuid: "170ecc40857b446f8034b62956e7f13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" is_pre_election: true
I20260809 20:24:29.760070 20938 raft_consensus.cc:2454] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170ecc40857b446f8034b62956e7f13b in term 0.
I20260809 20:24:29.760385 20790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "706dd42119594228adaf900ea3f24afe" candidate_uuid: "170ecc40857b446f8034b62956e7f13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e681dee7227443c5a6916bf22a8bcd46" is_pre_election: true
I20260809 20:24:29.761080 20790 raft_consensus.cc:2454] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170ecc40857b446f8034b62956e7f13b in term 0.
I20260809 20:24:29.761060 21007 leader_election.cc:302] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170ecc40857b446f8034b62956e7f13b, ba56fdec4011408c8d8bc08e7d4e33e1; no voters:
I20260809 20:24:29.761575 21163 raft_consensus.cc:2781] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:24:29.761763 21163 raft_consensus.cc:482] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:24:29.761888 21163 raft_consensus.cc:3037] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:29.764104 21163 raft_consensus.cc:504] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } }
I20260809 20:24:29.764739 21163 leader_election.cc:288] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [CANDIDATE]: Term 1 election: Requested vote from peers ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2:37333), e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1:41219)
I20260809 20:24:29.765250 20938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "706dd42119594228adaf900ea3f24afe" candidate_uuid: "170ecc40857b446f8034b62956e7f13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1"
I20260809 20:24:29.765250 20790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "706dd42119594228adaf900ea3f24afe" candidate_uuid: "170ecc40857b446f8034b62956e7f13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e681dee7227443c5a6916bf22a8bcd46"
I20260809 20:24:29.765461 20938 raft_consensus.cc:3037] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:29.765461 20790 raft_consensus.cc:3037] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:29.768766 20790 raft_consensus.cc:2454] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170ecc40857b446f8034b62956e7f13b in term 1.
I20260809 20:24:29.768767 20938 raft_consensus.cc:2454] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170ecc40857b446f8034b62956e7f13b in term 1.
I20260809 20:24:29.769205 21007 leader_election.cc:302] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170ecc40857b446f8034b62956e7f13b, ba56fdec4011408c8d8bc08e7d4e33e1; no voters:
I20260809 20:24:29.769544 21163 raft_consensus.cc:2781] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:29.770006 21163 raft_consensus.cc:686] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [term 1 LEADER]: Becoming Leader. State: Replica: 170ecc40857b446f8034b62956e7f13b, State: Running, Role: LEADER
I20260809 20:24:29.770467 21163 consensus_queue.cc:237] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } }
I20260809 20:24:29.775645 20619 catalog_manager.cc:5495] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b reported cstate change: term changed from 0 to 1, leader changed from <none> to 170ecc40857b446f8034b62956e7f13b (127.12.78.3). New cstate: current_term: 1 leader_uuid: "170ecc40857b446f8034b62956e7f13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } health_report { overall_health: UNKNOWN } } }
I20260809 20:24:29.802826 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:29.797303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.12.78.3:41445)
0809 20:24:29.797468 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:29.797473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:29.797525 (+ 52us) server_negotiation.cc:407] Connection header received
0809 20:24:29.797777 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:29.797782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:29.797869 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:29.798048 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:24:29.798320 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.799130 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.800496 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:29.801152 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:29.801299 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:29.801661 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:24:29.802220 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:24:29.802325 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:29.802436 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 20:24:29.802498 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 20:24:29.810742 20938 raft_consensus.cc:1261] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Refusing update from remote peer 170ecc40857b446f8034b62956e7f13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:29.810748 20790 raft_consensus.cc:1261] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Refusing update from remote peer 170ecc40857b446f8034b62956e7f13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:29.811662 21165 consensus_queue.cc:1035] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:29.811980 21163 consensus_queue.cc:1035] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:29.823819 21168 mvcc.cc:205] Tried to move back new op lower bound from 7316713757935255552 to 7316713757783826432. Current Snapshot: MvccSnapshot[applied={T|T < 7316713757935255552}]
I20260809 20:24:29.837988 20619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:29.838287 20619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:29.842253 20619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49: Sending DeleteTablet for 3 replicas of tablet 706dd42119594228adaf900ea3f24afe
I20260809 20:24:29.843389 21057 tablet_service.cc:1514] Processing DeleteTablet for tablet 706dd42119594228adaf900ea3f24afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:29 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:43366
I20260809 20:24:29.843443 20764 tablet_service.cc:1514] Processing DeleteTablet for tablet 706dd42119594228adaf900ea3f24afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:29 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:33392
I20260809 20:24:29.843426 20912 tablet_service.cc:1514] Processing DeleteTablet for tablet 706dd42119594228adaf900ea3f24afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:29 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:42986
I20260809 20:24:29.848369 20619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:24:29.849114 20619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:24:29.857527 21179 tablet_replica.cc:330] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1: stopping tablet replica
I20260809 20:24:29.858232 21180 tablet_replica.cc:330] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46: stopping tablet replica
I20260809 20:24:29.858844 21180 raft_consensus.cc:2229] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:29.859182 21056 tablet_service.cc:1467] Processing CreateTablet for tablet 23c6e3defbb545d4ba9af9d0af1b142c (DEFAULT_TABLE table=test-table [id=762662c9c42247bfa864641348ae59d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:29.859548 21179 raft_consensus.cc:2229] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:29.859781 21056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c6e3defbb545d4ba9af9d0af1b142c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:29.860358 21180 raft_consensus.cc:2258] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:29.861034 21179 raft_consensus.cc:2258] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:29.859592 20764 tablet_service.cc:1467] Processing CreateTablet for tablet 23c6e3defbb545d4ba9af9d0af1b142c (DEFAULT_TABLE table=test-table [id=762662c9c42247bfa864641348ae59d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:29.861745 20764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c6e3defbb545d4ba9af9d0af1b142c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:29.869277 21178 tablet_replica.cc:330] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b: stopping tablet replica
I20260809 20:24:29.869781 21178 raft_consensus.cc:2229] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:24:29.869769 20912 tablet_service.cc:1467] Processing CreateTablet for tablet 23c6e3defbb545d4ba9af9d0af1b142c (DEFAULT_TABLE table=test-table [id=762662c9c42247bfa864641348ae59d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:29.870241 20912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c6e3defbb545d4ba9af9d0af1b142c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:29.870559 21178 raft_consensus.cc:2258] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:29.872018 21180 ts_tablet_manager.cc:1905] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:29.872018 21179 ts_tablet_manager.cc:1905] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:29.874943 21157 tablet_bootstrap.cc:492] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b: Bootstrap starting.
I20260809 20:24:29.876726 21157 tablet_bootstrap.cc:654] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:29.879087 21180 ts_tablet_manager.cc:1918] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:29.879268 21180 log.cc:1196] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/706dd42119594228adaf900ea3f24afe
I20260809 20:24:29.879846 21180 ts_tablet_manager.cc:1939] T 706dd42119594228adaf900ea3f24afe P e681dee7227443c5a6916bf22a8bcd46: Deleting consensus metadata
I20260809 20:24:29.881791 20602 catalog_manager.cc:4841] TS e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1:41219): tablet 706dd42119594228adaf900ea3f24afe (table test-table [id=f5fd53974b6d46678b2e329fe5b7f099]) successfully deleted
I20260809 20:24:29.882699 21178 ts_tablet_manager.cc:1905] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:29.884261 21154 tablet_bootstrap.cc:492] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46: Bootstrap starting.
I20260809 20:24:29.886025 21154 tablet_bootstrap.cc:654] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:29.888029 21179 ts_tablet_manager.cc:1918] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:29.888236 21179 log.cc:1196] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/706dd42119594228adaf900ea3f24afe
I20260809 20:24:29.888947 21179 ts_tablet_manager.cc:1939] T 706dd42119594228adaf900ea3f24afe P ba56fdec4011408c8d8bc08e7d4e33e1: Deleting consensus metadata
I20260809 20:24:29.891191 20604 catalog_manager.cc:4841] TS ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2:37333): tablet 706dd42119594228adaf900ea3f24afe (table test-table [id=f5fd53974b6d46678b2e329fe5b7f099]) successfully deleted
I20260809 20:24:29.891350 21156 tablet_bootstrap.cc:492] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1: Bootstrap starting.
I20260809 20:24:29.893329 21156 tablet_bootstrap.cc:654] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:29.894181 21157 tablet_bootstrap.cc:492] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b: No bootstrap required, opened a new log
I20260809 20:24:29.894349 21154 tablet_bootstrap.cc:492] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46: No bootstrap required, opened a new log
I20260809 20:24:29.894390 21157 ts_tablet_manager.cc:1397] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.005s
I20260809 20:24:29.894528 21154 ts_tablet_manager.cc:1397] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260809 20:24:29.895447 21154 raft_consensus.cc:348] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:29.895679 21154 raft_consensus.cc:374] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:29.895779 21154 raft_consensus.cc:729] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e681dee7227443c5a6916bf22a8bcd46, State: Initialized, Role: FOLLOWER
I20260809 20:24:29.896121 21154 consensus_queue.cc:260] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:29.896874 21154 ts_tablet_manager.cc:1428] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260809 20:24:29.901638 21157 raft_consensus.cc:348] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:29.902070 21157 raft_consensus.cc:374] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:29.902623 21157 raft_consensus.cc:729] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170ecc40857b446f8034b62956e7f13b, State: Initialized, Role: FOLLOWER
I20260809 20:24:29.904637 21157 consensus_queue.cc:260] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:29.905583 21157 ts_tablet_manager.cc:1428] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260809 20:24:29.908134 21178 ts_tablet_manager.cc:1918] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:29.908344 21178 log.cc:1196] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/706dd42119594228adaf900ea3f24afe
I20260809 20:24:29.908955 21178 ts_tablet_manager.cc:1939] T 706dd42119594228adaf900ea3f24afe P 170ecc40857b446f8034b62956e7f13b: Deleting consensus metadata
I20260809 20:24:29.910746 20605 catalog_manager.cc:4841] TS 170ecc40857b446f8034b62956e7f13b (127.12.78.3:41445): tablet 706dd42119594228adaf900ea3f24afe (table test-table [id=f5fd53974b6d46678b2e329fe5b7f099]) successfully deleted
I20260809 20:24:29.912812 21156 tablet_bootstrap.cc:492] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1: No bootstrap required, opened a new log
I20260809 20:24:29.913043 21156 ts_tablet_manager.cc:1397] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.006s
I20260809 20:24:29.914000 21156 raft_consensus.cc:348] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:29.914263 21156 raft_consensus.cc:374] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:29.914376 21156 raft_consensus.cc:729] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba56fdec4011408c8d8bc08e7d4e33e1, State: Initialized, Role: FOLLOWER
I20260809 20:24:29.914652 21156 consensus_queue.cc:260] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:29.915323 21156 ts_tablet_manager.cc:1428] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260809 20:24:30.091290 20838 tablet.cc:2369] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:24:30.169634 21173 raft_consensus.cc:482] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:24:30.169893 21173 raft_consensus.cc:504] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.170830 21173 leader_election.cc:288] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1:41219), ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2:37333)
I20260809 20:24:30.171401 20790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c6e3defbb545d4ba9af9d0af1b142c" candidate_uuid: "170ecc40857b446f8034b62956e7f13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e681dee7227443c5a6916bf22a8bcd46" is_pre_election: true
I20260809 20:24:30.171532 20938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c6e3defbb545d4ba9af9d0af1b142c" candidate_uuid: "170ecc40857b446f8034b62956e7f13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" is_pre_election: true
I20260809 20:24:30.171697 20790 raft_consensus.cc:2454] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170ecc40857b446f8034b62956e7f13b in term 0.
I20260809 20:24:30.171789 20938 raft_consensus.cc:2454] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170ecc40857b446f8034b62956e7f13b in term 0.
I20260809 20:24:30.172214 21005 leader_election.cc:302] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170ecc40857b446f8034b62956e7f13b, e681dee7227443c5a6916bf22a8bcd46; no voters:
I20260809 20:24:30.172591 21173 raft_consensus.cc:2781] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:24:30.172747 21173 raft_consensus.cc:482] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:24:30.172851 21173 raft_consensus.cc:3037] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:30.175082 21173 raft_consensus.cc:504] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.175719 21173 leader_election.cc:288] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [CANDIDATE]: Term 1 election: Requested vote from peers e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1:41219), ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2:37333)
I20260809 20:24:30.176231 20790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c6e3defbb545d4ba9af9d0af1b142c" candidate_uuid: "170ecc40857b446f8034b62956e7f13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e681dee7227443c5a6916bf22a8bcd46"
I20260809 20:24:30.176277 20938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c6e3defbb545d4ba9af9d0af1b142c" candidate_uuid: "170ecc40857b446f8034b62956e7f13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1"
I20260809 20:24:30.176457 20790 raft_consensus.cc:3037] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:30.176476 20938 raft_consensus.cc:3037] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:30.179494 20790 raft_consensus.cc:2454] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170ecc40857b446f8034b62956e7f13b in term 1.
I20260809 20:24:30.179504 20938 raft_consensus.cc:2454] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170ecc40857b446f8034b62956e7f13b in term 1.
I20260809 20:24:30.179991 21007 leader_election.cc:302] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170ecc40857b446f8034b62956e7f13b, ba56fdec4011408c8d8bc08e7d4e33e1; no voters:
I20260809 20:24:30.180315 21173 raft_consensus.cc:2781] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:30.180573 21173 raft_consensus.cc:686] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [term 1 LEADER]: Becoming Leader. State: Replica: 170ecc40857b446f8034b62956e7f13b, State: Running, Role: LEADER
I20260809 20:24:30.180899 21173 consensus_queue.cc:237] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.183902 20618 catalog_manager.cc:5495] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b reported cstate change: term changed from 0 to 1, leader changed from <none> to 170ecc40857b446f8034b62956e7f13b (127.12.78.3). New cstate: current_term: 1 leader_uuid: "170ecc40857b446f8034b62956e7f13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } health_report { overall_health: HEALTHY } } }
Aug 09 20:24:30 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307067, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.62@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
I20260809 20:24:30.293465 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:30.275048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44837 (local address 127.0.0.1:51200)
0809 20:24:30.275778 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:30.275801 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:30.275937 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:30.276639 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:30.276644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:30.277330 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:30.277558 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:30.277567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.278620 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.278623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:30.279592 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:30.279604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.280293 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.280297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:30.280428 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:30.285270 (+ 4842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:30.285293 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:30.288017 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:30.291020 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:30.291032 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:30.291039 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:30.291496 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:30.291847 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:30.291851 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:30.291853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:30.292017 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:30.292512 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:30.292519 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:30.292742 (+ 223us) client_negotiation.cc:769] Sending connection context
0809 20:24:30.292889 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 20:24:30.293125 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":438,"threads_started":1}
I20260809 20:24:30.293608 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:30.275075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.12.78.62:44837)
0809 20:24:30.276210 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:30.276215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:30.276234 (+ 19us) server_negotiation.cc:407] Connection header received
0809 20:24:30.276280 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:30.276283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:30.276337 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:30.276430 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:30.277721 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.278489 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.279734 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.280137 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.280230 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:30.288244 (+ 8014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:30.288250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:30.288253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:30.288284 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:30.290860 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:30.291605 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:30.291609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:30.291611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:30.291739 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:30.292109 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:30.292121 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:30.292123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:30.292375 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:30.292501 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:30.292972 (+ 471us) server_negotiation.cc:299] Negotiation successful
0809 20:24:30.293261 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":1043,"thread_start_us":852,"threads_started":1}
I20260809 20:24:30.298769 20618 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:51200:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 20:24:30.299906 20618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:24:30.306952 20764 tablet_service.cc:1467] Processing CreateTablet for tablet eb8cd02e1a114bcfb079ec693685085c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=134f63ace4f943b79309e3f84fcb5846]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 20:24:30.307070 20912 tablet_service.cc:1467] Processing CreateTablet for tablet eb8cd02e1a114bcfb079ec693685085c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=134f63ace4f943b79309e3f84fcb5846]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 20:24:30.307629 20764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8cd02e1a114bcfb079ec693685085c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:30.307637 20912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8cd02e1a114bcfb079ec693685085c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:30.309108 21056 tablet_service.cc:1467] Processing CreateTablet for tablet eb8cd02e1a114bcfb079ec693685085c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=134f63ace4f943b79309e3f84fcb5846]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 20:24:30.309635 21056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8cd02e1a114bcfb079ec693685085c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:30.314774 21157 tablet_bootstrap.cc:492] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b: Bootstrap starting.
I20260809 20:24:30.315037 21156 tablet_bootstrap.cc:492] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1: Bootstrap starting.
I20260809 20:24:30.315066 21154 tablet_bootstrap.cc:492] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46: Bootstrap starting.
I20260809 20:24:30.316565 21157 tablet_bootstrap.cc:654] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:30.316745 21154 tablet_bootstrap.cc:654] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:30.316864 21156 tablet_bootstrap.cc:654] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:30.318974 21154 tablet_bootstrap.cc:492] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46: No bootstrap required, opened a new log
I20260809 20:24:30.319161 21154 ts_tablet_manager.cc:1397] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260809 20:24:30.319190 21156 tablet_bootstrap.cc:492] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1: No bootstrap required, opened a new log
I20260809 20:24:30.319343 21156 ts_tablet_manager.cc:1397] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 20:24:30.319542 21157 tablet_bootstrap.cc:492] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b: No bootstrap required, opened a new log
I20260809 20:24:30.319710 21157 ts_tablet_manager.cc:1397] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 20:24:30.319962 21154 raft_consensus.cc:348] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.320094 21156 raft_consensus.cc:348] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.320241 21154 raft_consensus.cc:374] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:30.320324 21154 raft_consensus.cc:729] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e681dee7227443c5a6916bf22a8bcd46, State: Initialized, Role: FOLLOWER
I20260809 20:24:30.320334 21156 raft_consensus.cc:374] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:30.320422 21156 raft_consensus.cc:729] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba56fdec4011408c8d8bc08e7d4e33e1, State: Initialized, Role: FOLLOWER
I20260809 20:24:30.320431 21157 raft_consensus.cc:348] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.320706 21157 raft_consensus.cc:374] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:30.320606 21154 consensus_queue.cc:260] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.320784 21157 raft_consensus.cc:729] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170ecc40857b446f8034b62956e7f13b, State: Initialized, Role: FOLLOWER
I20260809 20:24:30.320705 21156 consensus_queue.cc:260] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.321152 21154 ts_tablet_manager.cc:1428] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260809 20:24:30.321290 21156 ts_tablet_manager.cc:1428] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 20:24:30.321182 21162 tablet_replica.cc:439] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } }
I20260809 20:24:30.321509 21162 tablet_replica.cc:442] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:24:30.321596 21161 tablet_replica.cc:439] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } }
I20260809 20:24:30.321724 21157 consensus_queue.cc:260] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.321928 21161 tablet_replica.cc:442] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:24:30.322299 21173 tablet_replica.cc:439] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } }
I20260809 20:24:30.322583 21173 tablet_replica.cc:442] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:24:30.322634 21157 ts_tablet_manager.cc:1428] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 20:24:30.362488 21173 raft_consensus.cc:482] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:24:30.362739 21173 raft_consensus.cc:504] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.363507 21173 leader_election.cc:288] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1:41219), ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2:37333)
I20260809 20:24:30.364045 20790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8cd02e1a114bcfb079ec693685085c" candidate_uuid: "170ecc40857b446f8034b62956e7f13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e681dee7227443c5a6916bf22a8bcd46" is_pre_election: true
I20260809 20:24:30.364143 20938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8cd02e1a114bcfb079ec693685085c" candidate_uuid: "170ecc40857b446f8034b62956e7f13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" is_pre_election: true
I20260809 20:24:30.364353 20790 raft_consensus.cc:2454] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170ecc40857b446f8034b62956e7f13b in term 0.
I20260809 20:24:30.364385 20938 raft_consensus.cc:2454] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170ecc40857b446f8034b62956e7f13b in term 0.
I20260809 20:24:30.364831 21005 leader_election.cc:302] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170ecc40857b446f8034b62956e7f13b, e681dee7227443c5a6916bf22a8bcd46; no voters:
I20260809 20:24:30.365132 21173 raft_consensus.cc:2781] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:24:30.365221 21173 raft_consensus.cc:482] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:24:30.365365 21173 raft_consensus.cc:3037] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:30.367295 21173 raft_consensus.cc:504] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.367849 21173 leader_election.cc:288] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [CANDIDATE]: Term 1 election: Requested vote from peers e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1:41219), ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2:37333)
I20260809 20:24:30.368284 20790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8cd02e1a114bcfb079ec693685085c" candidate_uuid: "170ecc40857b446f8034b62956e7f13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e681dee7227443c5a6916bf22a8bcd46"
I20260809 20:24:30.368296 20938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8cd02e1a114bcfb079ec693685085c" candidate_uuid: "170ecc40857b446f8034b62956e7f13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1"
I20260809 20:24:30.368499 20790 raft_consensus.cc:3037] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:30.368510 20938 raft_consensus.cc:3037] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:30.370442 20790 raft_consensus.cc:2454] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170ecc40857b446f8034b62956e7f13b in term 1.
I20260809 20:24:30.370450 20938 raft_consensus.cc:2454] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170ecc40857b446f8034b62956e7f13b in term 1.
I20260809 20:24:30.370844 21007 leader_election.cc:302] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170ecc40857b446f8034b62956e7f13b, ba56fdec4011408c8d8bc08e7d4e33e1; no voters:
I20260809 20:24:30.371129 21173 raft_consensus.cc:2781] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:30.371321 21173 raft_consensus.cc:686] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [term 1 LEADER]: Becoming Leader. State: Replica: 170ecc40857b446f8034b62956e7f13b, State: Running, Role: LEADER
I20260809 20:24:30.371587 21173 consensus_queue.cc:237] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.373199 21165 tablet_replica.cc:439] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b: TxnStatusTablet state changed. Reason: New leader 170ecc40857b446f8034b62956e7f13b. Latest consensus state: current_term: 1 leader_uuid: "170ecc40857b446f8034b62956e7f13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } }
I20260809 20:24:30.373447 21165 tablet_replica.cc:442] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b: This TxnStatusTablet replica's current role is: LEADER
I20260809 20:24:30.374018 21199 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 20:24:30.374248 21199 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 20:24:30.374745 20616 catalog_manager.cc:5495] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b reported cstate change: term changed from 0 to 1, leader changed from <none> to 170ecc40857b446f8034b62956e7f13b (127.12.78.3). New cstate: current_term: 1 leader_uuid: "170ecc40857b446f8034b62956e7f13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } health_report { overall_health: HEALTHY } } }
I20260809 20:24:30.406420 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:30.399929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.12.78.3:41445)
0809 20:24:30.400527 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:30.400532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:30.400599 (+ 67us) server_negotiation.cc:407] Connection header received
0809 20:24:30.400668 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:30.400671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:30.400731 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:30.400863 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:24:30.401307 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.402224 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.404206 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.404816 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.404931 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:30.405526 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:24:30.405660 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:24:30.405739 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:30.405919 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 20:24:30.406045 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":288,"threads_started":1}
I20260809 20:24:30.406427 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:30.399735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:41445 (local address 127.0.0.1:43374)
0809 20:24:30.399952 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:30.399969 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:30.400072 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:30.400924 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:30.400929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:30.400944 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 20:24:30.401148 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:30.401162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.402358 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.402364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:30.404022 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:30.404038 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.404986 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.404993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:30.405216 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:30.405295 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 20:24:30.405775 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 20:24:30.405798 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 20:24:30.405894 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 20:24:30.406062 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 20:24:30.410606 20938 raft_consensus.cc:1261] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Refusing update from remote peer 170ecc40857b446f8034b62956e7f13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:30.410632 20790 raft_consensus.cc:1261] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Refusing update from remote peer 170ecc40857b446f8034b62956e7f13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:30.411345 21165 consensus_queue.cc:1035] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:30.411761 21173 consensus_queue.cc:1035] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:30.418982 21161 tablet_replica.cc:439] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46: TxnStatusTablet state changed. Reason: New leader 170ecc40857b446f8034b62956e7f13b. Latest consensus state: current_term: 1 leader_uuid: "170ecc40857b446f8034b62956e7f13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } }
I20260809 20:24:30.419370 21161 tablet_replica.cc:442] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:24:30.422398 21204 mvcc.cc:205] Tried to move back new op lower bound from 7316713760395915264 to 7316713760244379648. Current Snapshot: MvccSnapshot[applied={T|T < 7316713760395915264}]
I20260809 20:24:30.424424 21161 tablet_replica.cc:439] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "170ecc40857b446f8034b62956e7f13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } }
I20260809 20:24:30.424728 21161 tablet_replica.cc:442] T eb8cd02e1a114bcfb079ec693685085c P e681dee7227443c5a6916bf22a8bcd46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:24:30.434473 21173 tablet_replica.cc:439] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "170ecc40857b446f8034b62956e7f13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } }
I20260809 20:24:30.434979 21173 tablet_replica.cc:442] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b: This TxnStatusTablet replica's current role is: LEADER
I20260809 20:24:30.438823 21162 tablet_replica.cc:439] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1: TxnStatusTablet state changed. Reason: New leader 170ecc40857b446f8034b62956e7f13b. Latest consensus state: current_term: 1 leader_uuid: "170ecc40857b446f8034b62956e7f13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } }
I20260809 20:24:30.439100 21162 tablet_replica.cc:442] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:24:30.441525 21173 tablet_replica.cc:439] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "170ecc40857b446f8034b62956e7f13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } }
I20260809 20:24:30.441817 21173 tablet_replica.cc:442] T eb8cd02e1a114bcfb079ec693685085c P 170ecc40857b446f8034b62956e7f13b: This TxnStatusTablet replica's current role is: LEADER
I20260809 20:24:30.444793 21162 tablet_replica.cc:439] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "170ecc40857b446f8034b62956e7f13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } } }
I20260809 20:24:30.445039 21162 tablet_replica.cc:442] T eb8cd02e1a114bcfb079ec693685085c P ba56fdec4011408c8d8bc08e7d4e33e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 20:24:30.460726 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:30.454778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:53295 (local address 127.12.78.3:41445)
0809 20:24:30.454945 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:30.454955 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:30.455604 (+ 649us) server_negotiation.cc:407] Connection header received
0809 20:24:30.455809 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:30.455814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:30.455885 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:30.456007 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:24:30.456691 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.457436 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.458916 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.459342 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.459484 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:30.459894 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:24:30.460010 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:24:30.460110 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:30.460260 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 20:24:30.460325 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 20:24:30.460813 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:30.454903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:41445 (local address 127.12.78.3:53295)
0809 20:24:30.455503 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:30.455520 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:30.455679 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:30.456179 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:30.456182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:30.456191 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 20:24:30.456523 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:30.456536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.457594 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.457602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:30.458770 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:30.458782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.459508 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.459513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:30.459669 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:30.459723 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 20:24:30.460109 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 20:24:30.460120 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 20:24:30.460241 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 20:24:30.460366 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":299,"threads_started":1}
I20260809 20:24:30.472352 20790 raft_consensus.cc:1261] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Refusing update from remote peer 170ecc40857b446f8034b62956e7f13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:30.472584 20937 raft_consensus.cc:1261] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Refusing update from remote peer 170ecc40857b446f8034b62956e7f13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:30.473194 21173 consensus_queue.cc:1035] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:30.473922 21165 consensus_queue.cc:1035] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:30.540215 20616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:30.540500 20616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:30.543689 20616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49: Sending DeleteTablet for 3 replicas of tablet 23c6e3defbb545d4ba9af9d0af1b142c
I20260809 20:24:30.544525 20764 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c6e3defbb545d4ba9af9d0af1b142c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:30 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:33392
I20260809 20:24:30.544626 20912 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c6e3defbb545d4ba9af9d0af1b142c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:30 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:42986
I20260809 20:24:30.549242 21056 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c6e3defbb545d4ba9af9d0af1b142c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:30 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:43366
I20260809 20:24:30.565258 21221 tablet_replica.cc:330] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b: stopping tablet replica
I20260809 20:24:30.565318 21220 tablet_replica.cc:330] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1: stopping tablet replica
I20260809 20:24:30.565544 21220 raft_consensus.cc:2229] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:30.565868 21220 raft_consensus.cc:2258] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:30.565930 21221 raft_consensus.cc:2229] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:24:30.566666 21219 tablet_replica.cc:330] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46: stopping tablet replica
I20260809 20:24:30.566615 21221 raft_consensus.cc:2258] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:30.566906 21219 raft_consensus.cc:2229] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:30.567212 21219 raft_consensus.cc:2258] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:30.567216 21220 ts_tablet_manager.cc:1905] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:30.568142 21221 ts_tablet_manager.cc:1905] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:30.568528 21219 ts_tablet_manager.cc:1905] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:30.576647 21219 ts_tablet_manager.cc:1918] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:24:30.576817 21219 log.cc:1196] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/23c6e3defbb545d4ba9af9d0af1b142c
I20260809 20:24:30.576865 21221 ts_tablet_manager.cc:1918] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:24:30.577196 21221 log.cc:1196] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/23c6e3defbb545d4ba9af9d0af1b142c
I20260809 20:24:30.577383 21219 ts_tablet_manager.cc:1939] T 23c6e3defbb545d4ba9af9d0af1b142c P e681dee7227443c5a6916bf22a8bcd46: Deleting consensus metadata
I20260809 20:24:30.577847 21221 ts_tablet_manager.cc:1939] T 23c6e3defbb545d4ba9af9d0af1b142c P 170ecc40857b446f8034b62956e7f13b: Deleting consensus metadata
I20260809 20:24:30.578773 20602 catalog_manager.cc:4841] TS e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1:41219): tablet 23c6e3defbb545d4ba9af9d0af1b142c (table test-table [id=762662c9c42247bfa864641348ae59d2]) successfully deleted
I20260809 20:24:30.579854 20605 catalog_manager.cc:4841] TS 170ecc40857b446f8034b62956e7f13b (127.12.78.3:41445): tablet 23c6e3defbb545d4ba9af9d0af1b142c (table test-table [id=762662c9c42247bfa864641348ae59d2]) successfully deleted
I20260809 20:24:30.580415 21220 ts_tablet_manager.cc:1918] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:24:30.580577 21220 log.cc:1196] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/23c6e3defbb545d4ba9af9d0af1b142c
I20260809 20:24:30.581054 21220 ts_tablet_manager.cc:1939] T 23c6e3defbb545d4ba9af9d0af1b142c P ba56fdec4011408c8d8bc08e7d4e33e1: Deleting consensus metadata
I20260809 20:24:30.582119 20604 catalog_manager.cc:4841] TS ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2:37333): tablet 23c6e3defbb545d4ba9af9d0af1b142c (table test-table [id=762662c9c42247bfa864641348ae59d2]) successfully deleted
Aug 09 20:24:30 dist-test-slave-rnrw krb5kdc[20571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 20:24:30 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:30 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 20:24:30.624177 21193 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:30.612048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.12.78.62:44837)
0809 20:24:30.612318 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:30.612325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:30.612709 (+ 384us) server_negotiation.cc:407] Connection header received
0809 20:24:30.612917 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:30.612921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:30.612990 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:30.613088 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:30.614910 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.615713 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.617597 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.618032 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.618134 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:30.623926 (+ 5792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51212: BlockingRecv error: recv got EOF from 127.0.0.1:51212 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 20:24:30 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM
I20260809 20:24:30.642460 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:30.627052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.12.78.62:44837)
0809 20:24:30.627291 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:30.627296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:30.627602 (+ 306us) server_negotiation.cc:407] Connection header received
0809 20:24:30.627756 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:30.627759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:30.627823 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:30.627907 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:30.629417 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.630141 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.631882 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.632280 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.632369 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:30.636968 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:30.636973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:30.636976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:30.637009 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:30.639777 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:30.640408 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:30.640415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:30.640419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:30.640613 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:30.641011 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:30.641016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:30.641020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:30.641296 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:30.641573 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:30.641871 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 20:24:30.642103 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260809 20:24:30.645974 20616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:24:30.646600 20616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:24:30.653983 20764 tablet_service.cc:1467] Processing CreateTablet for tablet d25f91748492481ab08fac697bc3c5c7 (DEFAULT_TABLE table=test-table [id=ca16ae2cc21640f3b9c75ff9c9a63602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:30.654199 20912 tablet_service.cc:1467] Processing CreateTablet for tablet d25f91748492481ab08fac697bc3c5c7 (DEFAULT_TABLE table=test-table [id=ca16ae2cc21640f3b9c75ff9c9a63602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:30.654469 20764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25f91748492481ab08fac697bc3c5c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:30.654629 20912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25f91748492481ab08fac697bc3c5c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:30.659461 21154 tablet_bootstrap.cc:492] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46: Bootstrap starting.
I20260809 20:24:30.660676 21056 tablet_service.cc:1467] Processing CreateTablet for tablet d25f91748492481ab08fac697bc3c5c7 (DEFAULT_TABLE table=test-table [id=ca16ae2cc21640f3b9c75ff9c9a63602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:30.661194 21056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25f91748492481ab08fac697bc3c5c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:30.661207 21154 tablet_bootstrap.cc:654] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:30.661504 21156 tablet_bootstrap.cc:492] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1: Bootstrap starting.
I20260809 20:24:30.663249 21156 tablet_bootstrap.cc:654] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:30.671653 21157 tablet_bootstrap.cc:492] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b: Bootstrap starting.
I20260809 20:24:30.673034 21156 tablet_bootstrap.cc:492] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1: No bootstrap required, opened a new log
I20260809 20:24:30.673202 21156 ts_tablet_manager.cc:1397] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260809 20:24:30.673427 21154 tablet_bootstrap.cc:492] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46: No bootstrap required, opened a new log
I20260809 20:24:30.673511 21157 tablet_bootstrap.cc:654] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:30.673588 21154 ts_tablet_manager.cc:1397] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260809 20:24:30.674098 21156 raft_consensus.cc:348] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.674407 21156 raft_consensus.cc:374] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:30.674528 21156 raft_consensus.cc:729] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba56fdec4011408c8d8bc08e7d4e33e1, State: Initialized, Role: FOLLOWER
I20260809 20:24:30.674489 21154 raft_consensus.cc:348] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.674780 21154 raft_consensus.cc:374] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:30.674886 21154 raft_consensus.cc:729] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e681dee7227443c5a6916bf22a8bcd46, State: Initialized, Role: FOLLOWER
I20260809 20:24:30.674852 21156 consensus_queue.cc:260] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.675129 21154 consensus_queue.cc:260] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.676144 21154 ts_tablet_manager.cc:1428] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260809 20:24:30.678546 21156 ts_tablet_manager.cc:1428] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260809 20:24:30.685760 21157 tablet_bootstrap.cc:492] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b: No bootstrap required, opened a new log
I20260809 20:24:30.685961 21157 ts_tablet_manager.cc:1397] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20260809 20:24:30.686870 21157 raft_consensus.cc:348] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.687160 21157 raft_consensus.cc:374] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:30.687288 21157 raft_consensus.cc:729] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170ecc40857b446f8034b62956e7f13b, State: Initialized, Role: FOLLOWER
I20260809 20:24:30.687546 21157 consensus_queue.cc:260] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.688040 21157 ts_tablet_manager.cc:1428] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 20:24:30.943650 21161 raft_consensus.cc:482] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:24:30.943912 21161 raft_consensus.cc:504] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.945503 21161 leader_election.cc:288] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2:37333), 170ecc40857b446f8034b62956e7f13b (127.12.78.3:41445)
I20260809 20:24:30.954334 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:30.945963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:37221 (local address 127.12.78.3:41445)
0809 20:24:30.946192 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:30.946202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:30.946858 (+ 656us) server_negotiation.cc:407] Connection header received
0809 20:24:30.947175 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:30.947183 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:30.947259 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:30.947308 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:30.948167 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.949323 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.952245 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.953145 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.953287 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:30.953409 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:30.953912 (+ 503us) server_negotiation.cc:299] Negotiation successful
0809 20:24:30.954060 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 20:24:30.954453 21241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:30.945832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:37333 (local address 127.12.78.1:53059)
0809 20:24:30.946934 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:30.946950 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:30.947038 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:30.947558 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:30.947563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:30.947604 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:30.948108 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:30.948128 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.949623 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.949629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:30.951974 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:30.951990 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.953476 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.953481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:30.953616 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:30.953654 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 20:24:30.953774 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 20:24:30.953916 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":121,"thread_start_us":806,"threads_started":1}
I20260809 20:24:30.954474 21239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:30.945804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:41445 (local address 127.12.78.1:37221)
0809 20:24:30.946735 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:30.946756 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:30.946983 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:30.947469 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:30.947472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:30.947516 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:30.947977 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:30.947991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.949447 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.949451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:30.951974 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:30.951990 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.953548 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.953553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:30.953729 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:30.953768 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 20:24:30.953843 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 20:24:30.953931 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":63,"thread_start_us":559,"threads_started":1}
I20260809 20:24:30.954861 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:30.945897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:53059 (local address 127.12.78.2:37333)
0809 20:24:30.947281 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:30.947286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:30.947302 (+ 16us) server_negotiation.cc:407] Connection header received
0809 20:24:30.947359 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:30.947363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:30.947418 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:30.947520 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:30.948313 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.949492 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.952159 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:30.953098 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:30.953269 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:30.953371 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:30.954011 (+ 640us) server_negotiation.cc:299] Negotiation successful
0809 20:24:30.954138 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1275,"thread_start_us":1149,"threads_started":1}
I20260809 20:24:30.955361 20937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f91748492481ab08fac697bc3c5c7" candidate_uuid: "e681dee7227443c5a6916bf22a8bcd46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" is_pre_election: true
I20260809 20:24:30.955513 21083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f91748492481ab08fac697bc3c5c7" candidate_uuid: "e681dee7227443c5a6916bf22a8bcd46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170ecc40857b446f8034b62956e7f13b" is_pre_election: true
I20260809 20:24:30.955767 20937 raft_consensus.cc:2454] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e681dee7227443c5a6916bf22a8bcd46 in term 0.
I20260809 20:24:30.955842 21083 raft_consensus.cc:2454] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e681dee7227443c5a6916bf22a8bcd46 in term 0.
I20260809 20:24:30.956331 20712 leader_election.cc:302] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba56fdec4011408c8d8bc08e7d4e33e1, e681dee7227443c5a6916bf22a8bcd46; no voters:
I20260809 20:24:30.956750 21161 raft_consensus.cc:2781] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:24:30.956880 21161 raft_consensus.cc:482] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:24:30.956979 21161 raft_consensus.cc:3037] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:30.959033 21161 raft_consensus.cc:504] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.959568 21161 leader_election.cc:288] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [CANDIDATE]: Term 1 election: Requested vote from peers ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2:37333), 170ecc40857b446f8034b62956e7f13b (127.12.78.3:41445)
I20260809 20:24:30.959947 20937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f91748492481ab08fac697bc3c5c7" candidate_uuid: "e681dee7227443c5a6916bf22a8bcd46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1"
I20260809 20:24:30.960083 21083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f91748492481ab08fac697bc3c5c7" candidate_uuid: "e681dee7227443c5a6916bf22a8bcd46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170ecc40857b446f8034b62956e7f13b"
I20260809 20:24:30.960189 20937 raft_consensus.cc:3037] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:30.960309 21083 raft_consensus.cc:3037] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:30.963189 20937 raft_consensus.cc:2454] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e681dee7227443c5a6916bf22a8bcd46 in term 1.
I20260809 20:24:30.963356 21083 raft_consensus.cc:2454] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e681dee7227443c5a6916bf22a8bcd46 in term 1.
I20260809 20:24:30.963692 20712 leader_election.cc:302] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba56fdec4011408c8d8bc08e7d4e33e1, e681dee7227443c5a6916bf22a8bcd46; no voters:
I20260809 20:24:30.964077 21161 raft_consensus.cc:2781] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:30.964609 21161 raft_consensus.cc:686] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [term 1 LEADER]: Becoming Leader. State: Replica: e681dee7227443c5a6916bf22a8bcd46, State: Running, Role: LEADER
I20260809 20:24:30.964973 21161 consensus_queue.cc:237] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:30.968281 20619 catalog_manager.cc:5495] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 reported cstate change: term changed from 0 to 1, leader changed from <none> to e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1). New cstate: current_term: 1 leader_uuid: "e681dee7227443c5a6916bf22a8bcd46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } health_report { overall_health: UNKNOWN } } }
I20260809 20:24:31.045317 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:31.040039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.12.78.1:41219)
0809 20:24:31.040659 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:31.040664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:31.040681 (+ 17us) server_negotiation.cc:407] Connection header received
0809 20:24:31.040771 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:31.040775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:31.040850 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:31.040993 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:24:31.041323 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.042044 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.043292 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.043679 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.043767 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:31.044124 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:24:31.044743 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:24:31.044858 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:31.044949 (+ 91us) server_negotiation.cc:299] Negotiation successful
0809 20:24:31.045016 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":385,"threads_started":1}
I20260809 20:24:31.049265 20937 raft_consensus.cc:1261] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Refusing update from remote peer e681dee7227443c5a6916bf22a8bcd46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:31.049479 21083 raft_consensus.cc:1261] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Refusing update from remote peer e681dee7227443c5a6916bf22a8bcd46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:31.050096 21242 consensus_queue.cc:1035] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:31.050455 21161 consensus_queue.cc:1035] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:31.058697 21247 mvcc.cc:205] Tried to move back new op lower bound from 7316713763010961408 to 7316713762676330496. Current Snapshot: MvccSnapshot[applied={T|T < 7316713763010961408 or (T in {7316713763010961408})}]
I20260809 20:24:31.076264 20619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:31.076470 20619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:31.079907 20619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49: Sending DeleteTablet for 3 replicas of tablet d25f91748492481ab08fac697bc3c5c7
I20260809 20:24:31.080885 20912 tablet_service.cc:1514] Processing DeleteTablet for tablet d25f91748492481ab08fac697bc3c5c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:31 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:42986
I20260809 20:24:31.080991 20764 tablet_service.cc:1514] Processing DeleteTablet for tablet d25f91748492481ab08fac697bc3c5c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:31 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:33392
I20260809 20:24:31.081403 21220 tablet_replica.cc:330] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1: stopping tablet replica
I20260809 20:24:31.081368 21056 tablet_service.cc:1514] Processing DeleteTablet for tablet d25f91748492481ab08fac697bc3c5c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:31 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:43366
I20260809 20:24:31.081635 21220 raft_consensus.cc:2229] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:31.081960 21220 raft_consensus.cc:2258] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:31.083660 21220 ts_tablet_manager.cc:1905] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:31.083587 20619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:24:31.085198 20619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:24:31.092337 21220 ts_tablet_manager.cc:1918] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:31.092531 21220 log.cc:1196] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/d25f91748492481ab08fac697bc3c5c7
I20260809 20:24:31.093474 21220 ts_tablet_manager.cc:1939] T d25f91748492481ab08fac697bc3c5c7 P ba56fdec4011408c8d8bc08e7d4e33e1: Deleting consensus metadata
I20260809 20:24:31.094960 20763 tablet_service.cc:1467] Processing CreateTablet for tablet 241094f8c8df429f9df5b55bf5c4030a (DEFAULT_TABLE table=test-table [id=d0eb79a3110d4f9cacb61f0939b6e996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:31.095026 20912 tablet_service.cc:1467] Processing CreateTablet for tablet 241094f8c8df429f9df5b55bf5c4030a (DEFAULT_TABLE table=test-table [id=d0eb79a3110d4f9cacb61f0939b6e996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:31.095486 20763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 241094f8c8df429f9df5b55bf5c4030a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:31.095574 20912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 241094f8c8df429f9df5b55bf5c4030a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:31.096747 21057 tablet_service.cc:1467] Processing CreateTablet for tablet 241094f8c8df429f9df5b55bf5c4030a (DEFAULT_TABLE table=test-table [id=d0eb79a3110d4f9cacb61f0939b6e996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:24:31.097383 21057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 241094f8c8df429f9df5b55bf5c4030a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:31.105269 20604 catalog_manager.cc:4841] TS ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2:37333): tablet d25f91748492481ab08fac697bc3c5c7 (table test-table [id=ca16ae2cc21640f3b9c75ff9c9a63602]) successfully deleted
I20260809 20:24:31.107136 21156 tablet_bootstrap.cc:492] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1: Bootstrap starting.
I20260809 20:24:31.109481 21156 tablet_bootstrap.cc:654] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:31.112155 21157 tablet_bootstrap.cc:492] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b: Bootstrap starting.
I20260809 20:24:31.114095 21157 tablet_bootstrap.cc:654] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:31.124612 21257 tablet_replica.cc:330] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b: stopping tablet replica
I20260809 20:24:31.124857 21257 raft_consensus.cc:2229] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:31.125170 21257 raft_consensus.cc:2258] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:31.125622 21156 tablet_bootstrap.cc:492] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1: No bootstrap required, opened a new log
I20260809 20:24:31.126190 21156 ts_tablet_manager.cc:1397] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.003s
I20260809 20:24:31.126422 21257 ts_tablet_manager.cc:1905] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:31.127084 21156 raft_consensus.cc:348] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:31.127367 21156 raft_consensus.cc:374] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:31.127498 21156 raft_consensus.cc:729] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba56fdec4011408c8d8bc08e7d4e33e1, State: Initialized, Role: FOLLOWER
I20260809 20:24:31.127790 21156 consensus_queue.cc:260] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:31.128651 21157 tablet_bootstrap.cc:492] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b: No bootstrap required, opened a new log
I20260809 20:24:31.128816 21256 tablet_replica.cc:330] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46: stopping tablet replica
I20260809 20:24:31.128839 21157 ts_tablet_manager.cc:1397] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.010s
I20260809 20:24:31.129073 21156 ts_tablet_manager.cc:1428] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 20:24:31.129398 21154 tablet_bootstrap.cc:492] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46: Bootstrap starting.
I20260809 20:24:31.129693 21157 raft_consensus.cc:348] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:31.129935 21157 raft_consensus.cc:374] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:31.130014 21157 raft_consensus.cc:729] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170ecc40857b446f8034b62956e7f13b, State: Initialized, Role: FOLLOWER
I20260809 20:24:31.130265 21157 consensus_queue.cc:260] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:31.131505 21154 tablet_bootstrap.cc:654] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:31.131834 21157 ts_tablet_manager.cc:1428] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260809 20:24:31.129118 21256 raft_consensus.cc:2229] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:24:31.134475 21256 raft_consensus.cc:2258] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:31.136329 21256 ts_tablet_manager.cc:1905] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:31.140183 21257 ts_tablet_manager.cc:1918] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:31.140430 21257 log.cc:1196] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/d25f91748492481ab08fac697bc3c5c7
I20260809 20:24:31.141217 21257 ts_tablet_manager.cc:1939] T d25f91748492481ab08fac697bc3c5c7 P 170ecc40857b446f8034b62956e7f13b: Deleting consensus metadata
I20260809 20:24:31.142561 20605 catalog_manager.cc:4841] TS 170ecc40857b446f8034b62956e7f13b (127.12.78.3:41445): tablet d25f91748492481ab08fac697bc3c5c7 (table test-table [id=ca16ae2cc21640f3b9c75ff9c9a63602]) successfully deleted
I20260809 20:24:31.145495 21256 ts_tablet_manager.cc:1918] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:24:31.145728 21256 log.cc:1196] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/d25f91748492481ab08fac697bc3c5c7
I20260809 20:24:31.146229 21256 ts_tablet_manager.cc:1939] T d25f91748492481ab08fac697bc3c5c7 P e681dee7227443c5a6916bf22a8bcd46: Deleting consensus metadata
I20260809 20:24:31.146333 21154 tablet_bootstrap.cc:492] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46: No bootstrap required, opened a new log
I20260809 20:24:31.146580 21154 ts_tablet_manager.cc:1397] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20260809 20:24:31.147516 20602 catalog_manager.cc:4841] TS e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1:41219): tablet d25f91748492481ab08fac697bc3c5c7 (table test-table [id=ca16ae2cc21640f3b9c75ff9c9a63602]) successfully deleted
I20260809 20:24:31.147553 21154 raft_consensus.cc:348] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:31.147895 21154 raft_consensus.cc:374] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:31.148021 21154 raft_consensus.cc:729] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e681dee7227443c5a6916bf22a8bcd46, State: Initialized, Role: FOLLOWER
I20260809 20:24:31.148339 21154 consensus_queue.cc:260] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:31.148952 21154 ts_tablet_manager.cc:1428] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260809 20:24:31.399856 21162 raft_consensus.cc:482] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:24:31.400113 21162 raft_consensus.cc:504] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:31.401996 21162 leader_election.cc:288] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1:41219), 170ecc40857b446f8034b62956e7f13b (127.12.78.3:41445)
I20260809 20:24:31.409600 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:31.402474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:47189 (local address 127.12.78.3:41445)
0809 20:24:31.402738 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:31.402747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:31.403434 (+ 687us) server_negotiation.cc:407] Connection header received
0809 20:24:31.403735 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:31.403739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:31.403802 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:31.403905 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:31.404429 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.405352 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.407681 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.408615 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.408770 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:31.408876 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:31.409266 (+ 390us) server_negotiation.cc:299] Negotiation successful
0809 20:24:31.409389 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260809 20:24:31.410336 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:31.402268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:41445 (local address 127.12.78.2:47189)
0809 20:24:31.403305 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:31.403331 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:31.403567 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:31.403930 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:31.403935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:31.403961 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:31.404247 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:31.404256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.405528 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.405533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:31.407495 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:31.407511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.408853 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.408859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:31.409055 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:31.409097 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 20:24:31.409813 (+ 716us) client_negotiation.cc:240] Negotiation successful
0809 20:24:31.409956 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":1,"thread_start_us":1045,"threads_started":1}
I20260809 20:24:31.411073 21083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241094f8c8df429f9df5b55bf5c4030a" candidate_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170ecc40857b446f8034b62956e7f13b" is_pre_election: true
I20260809 20:24:31.411356 21083 raft_consensus.cc:2454] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba56fdec4011408c8d8bc08e7d4e33e1 in term 0.
I20260809 20:24:31.411547 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:31.402737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:57851 (local address 127.12.78.1:41219)
0809 20:24:31.402934 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:31.402941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:31.403700 (+ 759us) server_negotiation.cc:407] Connection header received
0809 20:24:31.403990 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:31.404001 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:31.404105 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:31.404175 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:24:31.405180 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.406342 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.409073 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.410471 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.410610 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:31.410731 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:31.411152 (+ 421us) server_negotiation.cc:299] Negotiation successful
0809 20:24:31.411288 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 20:24:31.411609 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:31.402285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:41219 (local address 127.12.78.2:57851)
0809 20:24:31.403587 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:31.403607 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:31.403812 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:31.404439 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:31.404443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:31.404473 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:24:31.404965 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:31.404981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.406574 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.406579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:31.408857 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:31.408874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.410764 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.410770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:31.410975 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:31.411005 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 20:24:31.411118 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 20:24:31.411243 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"mutex_wait_us":21,"thread_start_us":641,"threads_started":1}
I20260809 20:24:31.412099 20863 leader_election.cc:302] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170ecc40857b446f8034b62956e7f13b, ba56fdec4011408c8d8bc08e7d4e33e1; no voters:
I20260809 20:24:31.412381 20790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241094f8c8df429f9df5b55bf5c4030a" candidate_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e681dee7227443c5a6916bf22a8bcd46" is_pre_election: true
I20260809 20:24:31.412606 21162 raft_consensus.cc:2781] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:24:31.412652 20790 raft_consensus.cc:2454] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba56fdec4011408c8d8bc08e7d4e33e1 in term 0.
I20260809 20:24:31.412736 21162 raft_consensus.cc:482] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:24:31.412837 21162 raft_consensus.cc:3037] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:31.416193 21162 raft_consensus.cc:504] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:31.416949 21162 leader_election.cc:288] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [CANDIDATE]: Term 1 election: Requested vote from peers e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1:41219), 170ecc40857b446f8034b62956e7f13b (127.12.78.3:41445)
I20260809 20:24:31.417671 20790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241094f8c8df429f9df5b55bf5c4030a" candidate_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e681dee7227443c5a6916bf22a8bcd46"
I20260809 20:24:31.417676 21083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241094f8c8df429f9df5b55bf5c4030a" candidate_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170ecc40857b446f8034b62956e7f13b"
I20260809 20:24:31.417894 21083 raft_consensus.cc:3037] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:31.417893 20790 raft_consensus.cc:3037] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:31.422072 20790 raft_consensus.cc:2454] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba56fdec4011408c8d8bc08e7d4e33e1 in term 1.
I20260809 20:24:31.422705 20859 leader_election.cc:302] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba56fdec4011408c8d8bc08e7d4e33e1, e681dee7227443c5a6916bf22a8bcd46; no voters:
I20260809 20:24:31.422747 21083 raft_consensus.cc:2454] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba56fdec4011408c8d8bc08e7d4e33e1 in term 1.
I20260809 20:24:31.423554 21162 raft_consensus.cc:2781] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:31.424113 21162 raft_consensus.cc:686] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 LEADER]: Becoming Leader. State: Replica: ba56fdec4011408c8d8bc08e7d4e33e1, State: Running, Role: LEADER
I20260809 20:24:31.424718 21162 consensus_queue.cc:237] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } }
I20260809 20:24:31.431274 20616 catalog_manager.cc:5495] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2). New cstate: current_term: 1 leader_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba56fdec4011408c8d8bc08e7d4e33e1" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 } health_report { overall_health: UNKNOWN } } }
I20260809 20:24:31.498292 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:31.491546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.12.78.2:37333)
0809 20:24:31.492445 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:31.492451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:31.492472 (+ 21us) server_negotiation.cc:407] Connection header received
0809 20:24:31.492625 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:31.492631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:31.492742 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:31.492933 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:24:31.493321 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.494684 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.496020 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.496450 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.496637 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:31.497102 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:24:31.497674 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:24:31.497848 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:31.497901 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 20:24:31.497984 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":646,"threads_started":1}
I20260809 20:24:31.515750 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:31.509646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:41445 (local address 127.12.78.2:34077)
0809 20:24:31.510285 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:31.510305 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:31.510434 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:31.510794 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:31.510797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:31.510806 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 20:24:31.511108 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:31.511116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.512509 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.512513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:31.513639 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:31.513652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.514654 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.514659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:31.514776 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:31.514829 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 20:24:31.515267 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 20:24:31.515274 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 20:24:31.515335 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 20:24:31.515438 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":335,"threads_started":1}
I20260809 20:24:31.515811 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:31.509803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:34077 (local address 127.12.78.3:41445)
0809 20:24:31.510011 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:31.510018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:31.510405 (+ 387us) server_negotiation.cc:407] Connection header received
0809 20:24:31.510574 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:31.510579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:31.510674 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:31.510774 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:24:31.511270 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.512371 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.513786 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.514446 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.514594 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:31.515010 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:24:31.515163 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:24:31.515262 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:31.515390 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 20:24:31.515478 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 20:24:31.525162 20790 raft_consensus.cc:1261] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Refusing update from remote peer ba56fdec4011408c8d8bc08e7d4e33e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:31.525148 21083 raft_consensus.cc:1261] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Refusing update from remote peer ba56fdec4011408c8d8bc08e7d4e33e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:24:31.525960 21263 consensus_queue.cc:1035] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "170ecc40857b446f8034b62956e7f13b" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:31.526648 21162 consensus_queue.cc:1035] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e681dee7227443c5a6916bf22a8bcd46" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:24:31.566815 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:31.557912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:37333 (local address 127.12.78.3:55905)
0809 20:24:31.560158 (+ 2246us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:31.560179 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:31.560343 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:31.560911 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:31.560919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:31.560935 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 20:24:31.561665 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:31.561680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.563648 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.563654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:31.564796 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:31.564809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.565520 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.565527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:31.565680 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:31.565719 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 20:24:31.566164 (+ 445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 20:24:31.566178 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 20:24:31.566277 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 20:24:31.566429 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2083,"thread_start_us":1901,"threads_started":1}
I20260809 20:24:31.566921 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:31.558401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:55905 (local address 127.12.78.2:37333)
0809 20:24:31.559400 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:31.559409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:31.560291 (+ 882us) server_negotiation.cc:407] Connection header received
0809 20:24:31.560490 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:31.560497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:31.560616 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:31.560725 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:24:31.562297 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.563486 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.564942 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.565357 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.565503 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:31.565886 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:24:31.566034 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:24:31.566142 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:31.566571 (+ 429us) server_negotiation.cc:299] Negotiation successful
0809 20:24:31.566665 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799}
I20260809 20:24:31.605425 20616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:31.605741 20616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:24:31.609107 20616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4578952b05f448d3863f7aedae942f49: Sending DeleteTablet for 3 replicas of tablet 241094f8c8df429f9df5b55bf5c4030a
I20260809 20:24:31.609938 20764 tablet_service.cc:1514] Processing DeleteTablet for tablet 241094f8c8df429f9df5b55bf5c4030a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:31 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:33392
I20260809 20:24:31.610013 20912 tablet_service.cc:1514] Processing DeleteTablet for tablet 241094f8c8df429f9df5b55bf5c4030a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:31 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:42986
I20260809 20:24:31.610455 21256 tablet_replica.cc:330] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46: stopping tablet replica
I20260809 20:24:31.610456 21056 tablet_service.cc:1514] Processing DeleteTablet for tablet 241094f8c8df429f9df5b55bf5c4030a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:24:31 UTC) from {username='oryx', principal='oryx/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:43366
I20260809 20:24:31.610682 21256 raft_consensus.cc:2229] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:31.610752 21257 tablet_replica.cc:330] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b: stopping tablet replica
I20260809 20:24:31.610901 21256 raft_consensus.cc:2258] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:31.610936 21257 raft_consensus.cc:2229] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:24:31.611114 21257 raft_consensus.cc:2258] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:31.612287 21257 ts_tablet_manager.cc:1905] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:31.612285 21256 ts_tablet_manager.cc:1905] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:31.619004 21257 ts_tablet_manager.cc:1918] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:24:31.619004 21256 ts_tablet_manager.cc:1918] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:24:31.619168 21257 log.cc:1196] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/241094f8c8df429f9df5b55bf5c4030a
I20260809 20:24:31.619175 21256 log.cc:1196] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/241094f8c8df429f9df5b55bf5c4030a
I20260809 20:24:31.619587 21256 ts_tablet_manager.cc:1939] T 241094f8c8df429f9df5b55bf5c4030a P e681dee7227443c5a6916bf22a8bcd46: Deleting consensus metadata
I20260809 20:24:31.619593 21257 ts_tablet_manager.cc:1939] T 241094f8c8df429f9df5b55bf5c4030a P 170ecc40857b446f8034b62956e7f13b: Deleting consensus metadata
I20260809 20:24:31.620857 20602 catalog_manager.cc:4841] TS e681dee7227443c5a6916bf22a8bcd46 (127.12.78.1:41219): tablet 241094f8c8df429f9df5b55bf5c4030a (table test-table [id=d0eb79a3110d4f9cacb61f0939b6e996]) successfully deleted
I20260809 20:24:31.621483 20605 catalog_manager.cc:4841] TS 170ecc40857b446f8034b62956e7f13b (127.12.78.3:41445): tablet 241094f8c8df429f9df5b55bf5c4030a (table test-table [id=d0eb79a3110d4f9cacb61f0939b6e996]) successfully deleted
I20260809 20:24:31.622351 21278 tablet_replica.cc:330] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1: stopping tablet replica
I20260809 20:24:31.622396 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 20692
I20260809 20:24:31.622730 21278 raft_consensus.cc:2229] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:24:31.623371 21278 raft_consensus.cc:2258] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:24:31.625162 21278 ts_tablet_manager.cc:1905] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:24:31.633021 21278 ts_tablet_manager.cc:1918] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 20:24:31.633188 21278 log.cc:1196] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/241094f8c8df429f9df5b55bf5c4030a
I20260809 20:24:31.633671 21278 ts_tablet_manager.cc:1939] T 241094f8c8df429f9df5b55bf5c4030a P ba56fdec4011408c8d8bc08e7d4e33e1: Deleting consensus metadata
Aug 09 20:24:31 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.78.1@KRBTEST.COM
I20260809 20:24:31.634896 20604 catalog_manager.cc:4841] TS ba56fdec4011408c8d8bc08e7d4e33e1 (127.12.78.2:37333): tablet 241094f8c8df429f9df5b55bf5c4030a (table test-table [id=d0eb79a3110d4f9cacb61f0939b6e996]) successfully deleted
I20260809 20:24:31.641250 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:31.624781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.12.78.1:41219)
0809 20:24:31.625382 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:31.625391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:31.626155 (+ 764us) server_negotiation.cc:407] Connection header received
0809 20:24:31.626365 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:31.626371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:31.626442 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:31.626550 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:31.627734 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.628463 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.630034 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.630446 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.630584 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:31.635040 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:31.635062 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:31.635086 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:31.635137 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:31.638392 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:31.639095 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:31.639103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:31.639105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:31.639244 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:31.639653 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:31.639657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:31.639659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:31.640006 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:31.640343 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:31.640679 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 20:24:31.640920 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433}
I20260809 20:24:31.641995 20832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:31.782939 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 20692
I20260809 20:24:31.808719 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 20843
Aug 09 20:24:31 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.78.2@KRBTEST.COM
I20260809 20:24:31.826314 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:31.809520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.12.78.2:37333)
0809 20:24:31.809750 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:31.809756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:31.809784 (+ 28us) server_negotiation.cc:407] Connection header received
0809 20:24:31.809862 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:31.809872 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:31.809978 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:31.810108 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:31.811322 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.812655 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.813889 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.814640 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.814832 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:31.819160 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:31.819173 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:31.819192 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:31.819244 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:31.823313 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:31.823963 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:31.823973 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:31.823978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:31.824189 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:31.824568 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:31.824574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:31.824577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:31.825076 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:31.825455 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:31.825612 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 20:24:31.825855 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 20:24:31.827127 20980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:31.955551 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 20843
I20260809 20:24:31.978443 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 20989
Aug 09 20:24:31 dist-test-slave-rnrw krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.78.3@KRBTEST.COM
I20260809 20:24:31.994160 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:31.979355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.12.78.3:41445)
0809 20:24:31.979527 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:31.979531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:31.979548 (+ 17us) server_negotiation.cc:407] Connection header received
0809 20:24:31.979646 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:31.979650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:31.979713 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:31.979800 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:31.981268 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.982002 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.983223 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:31.983666 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:31.983774 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:31.988428 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:31.988445 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:31.988470 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:31.988509 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:31.991550 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:31.992244 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:31.992253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:31.992255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:31.992393 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:31.992837 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:31.992840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:31.992842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:31.993209 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:31.993445 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:31.993716 (+ 271us) server_negotiation.cc:299] Negotiation successful
0809 20:24:31.993902 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 20:24:31.994879 21124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:32.142382 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 20989
I20260809 20:24:32.171116 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 20586
I20260809 20:24:32.172382 20660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:32.291275 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 20586
2026-08-09T20:24:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6441 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:24:32 dist-test-slave-rnrw krb5kdc[21289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:24:32 dist-test-slave-rnrw krb5kdc[21289](info): set up 2 sockets
Aug 09 20:24:32 dist-test-slave-rnrw krb5kdc[21289](info): commencing operation
krb5kdc: starting...
W20260809 20:24:33.445261 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:24:33 dist-test-slave-rnrw krb5kdc[21289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:24:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:24:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:24:33.710147 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:39613
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:42943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:39613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:34.053879 21304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:34.054178 21304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:34.054348 21304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:34.061774 21304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:34.061892 21304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:34.061998 21304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:34.062107 21304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:34.062189 21304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:24:34.072274 21304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:42943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:39613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:39613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:34.073650 21304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:34.074982 21304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:34.087549 21310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:34.087621 21313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:34.088265 21311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:34.088271 21304 server_base.cc:1034] running on GCE node
I20260809 20:24:34.089156 21304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:34.090543 21304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:34.091761 21304 hybrid_clock.cc:648] HybridClock initialized: now 1786307074091714 us; error 35 us; skew 500 ppm
I20260809 20:24:34.092114 21304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T20:24:34Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1866 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:24:34 dist-test-slave-rnrw krb5kdc[21319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:24:34 dist-test-slave-rnrw krb5kdc[21319](info): set up 2 sockets
Aug 09 20:24:34 dist-test-slave-rnrw krb5kdc[21319](info): commencing operation
krb5kdc: starting...
W20260809 20:24:35.302611 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:24:35 dist-test-slave-rnrw krb5kdc[21319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:24:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:24:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:24:35.544173 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44301
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:44301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:35.868758 21333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:35.869046 21333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:35.869143 21333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:35.876302 21333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:35.876412 21333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:35.876470 21333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:35.876534 21333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:35.876645 21333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:24:35.886497 21333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:44301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:35.887776 21333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:35.889076 21333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:35.901167 21339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:35.901223 21342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:35.901240 21338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:35.902053 21333 server_base.cc:1034] running on GCE node
I20260809 20:24:35.902814 21333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:35.904181 21333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:35.905407 21333 hybrid_clock.cc:648] HybridClock initialized: now 1786307075905368 us; error 50 us; skew 500 ppm
I20260809 20:24:35.905786 21333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:35 dist-test-slave-rnrw krb5kdc[21319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307075, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:35.913931 21333 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:24:35.915459 21333 webserver.cc:460] Webserver started at http://127.12.78.62:32961/ using document root <none> and password file <none>
I20260809 20:24:35.916154 21333 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:35.916301 21333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:35.916689 21333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:35.919620 21333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "f41b1f518635464da64584a571a02701"
format_stamp: "Formatted at 2026-08-09 20:24:35 on dist-test-slave-rnrw"
server_key: "23dd6d8cdef30fce7bb718045fad6e1c"
server_key_iv: "94a0f3c18c0846cb4d700f12ed12e3fa"
server_key_version: "encryptionkey@0"
I20260809 20:24:35.920377 21333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "f41b1f518635464da64584a571a02701"
format_stamp: "Formatted at 2026-08-09 20:24:35 on dist-test-slave-rnrw"
server_key: "23dd6d8cdef30fce7bb718045fad6e1c"
server_key_iv: "94a0f3c18c0846cb4d700f12ed12e3fa"
server_key_version: "encryptionkey@0"
I20260809 20:24:35.925443 21333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 20:24:35.928977 21349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:35.930270 21333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 20:24:35.930444 21333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "f41b1f518635464da64584a571a02701"
format_stamp: "Formatted at 2026-08-09 20:24:35 on dist-test-slave-rnrw"
server_key: "23dd6d8cdef30fce7bb718045fad6e1c"
server_key_iv: "94a0f3c18c0846cb4d700f12ed12e3fa"
server_key_version: "encryptionkey@0"
I20260809 20:24:35.930650 21333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:35.962409 21333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:35.966508 21333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:35.966800 21333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:35.985498 21333 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:44301
I20260809 20:24:35.985587 21403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:44301 every 8 connection(s)
I20260809 20:24:35.986985 21333 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:24:35.990907 21404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:35.997067 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 21333
I20260809 20:24:35.997321 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:24:35.997996 12600 external_mini_cluster.cc:1428] Setting key 09f747a6f4d925e4519d322e75874436
I20260809 20:24:36.002803 21404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701: Bootstrap starting.
I20260809 20:24:36.006770 21404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:36.008098 21404 log.cc:824] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:36.011636 21404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701: No bootstrap required, opened a new log
Aug 09 20:24:36 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:24:36.019935 21404 raft_consensus.cc:348] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41b1f518635464da64584a571a02701" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44301 } }
I20260809 20:24:36.020327 21404 raft_consensus.cc:374] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:36.020473 21404 raft_consensus.cc:729] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41b1f518635464da64584a571a02701, State: Initialized, Role: FOLLOWER
I20260809 20:24:36.021155 21404 consensus_queue.cc:260] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41b1f518635464da64584a571a02701" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44301 } }
I20260809 20:24:36.021438 21404 raft_consensus.cc:388] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:24:36.021596 21404 raft_consensus.cc:482] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:24:36.021802 21404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:36.022413 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:36.000157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.12.78.62:44301)
0809 20:24:36.001088 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:36.001099 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:36.001143 (+ 44us) server_negotiation.cc:407] Connection header received
0809 20:24:36.002825 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:36.002848 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:36.003281 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:36.003624 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:36.005706 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:36.007131 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:36.009019 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:36.009917 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:36.010049 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:36.014742 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:36.014749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:36.014767 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:36.014808 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:36.018087 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:36.018826 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:36.018832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:36.018838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:36.019011 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:36.019418 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:36.019422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:36.019424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:36.019924 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:36.020255 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:36.020725 (+ 470us) server_negotiation.cc:299] Negotiation successful
0809 20:24:36.021189 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":461,"threads_started":1}
I20260809 20:24:36.024250 21404 raft_consensus.cc:504] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41b1f518635464da64584a571a02701" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44301 } }
I20260809 20:24:36.024858 21404 leader_election.cc:302] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f41b1f518635464da64584a571a02701; no voters:
I20260809 20:24:36.025552 21404 leader_election.cc:288] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:24:36.025650 21409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:36.026369 21409 raft_consensus.cc:686] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 1 LEADER]: Becoming Leader. State: Replica: f41b1f518635464da64584a571a02701, State: Running, Role: LEADER
I20260809 20:24:36.026894 21409 consensus_queue.cc:237] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41b1f518635464da64584a571a02701" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44301 } }
I20260809 20:24:36.027864 21404 sys_catalog.cc:564] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:36.030149 21410 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f41b1f518635464da64584a571a02701" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41b1f518635464da64584a571a02701" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44301 } } }
I20260809 20:24:36.030674 21410 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:36.030921 21411 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f41b1f518635464da64584a571a02701. Latest consensus state: current_term: 1 leader_uuid: "f41b1f518635464da64584a571a02701" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41b1f518635464da64584a571a02701" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44301 } } }
I20260809 20:24:36.031268 21411 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:36.032989 21415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:36.037802 21415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:36.045913 21415 catalog_manager.cc:1347] Generated new cluster ID: d82a35e28e6a4a1dab3951efb2ef92ba
I20260809 20:24:36.046020 21415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:36.047649 21427 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 20:24:36.055737 21415 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:24:36.057034 21415 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:36.065511 21415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701: Generated new TSK 0
I20260809 20:24:36.066286 21415 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:24:36.124243 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44301
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:36.465411 21431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:36.465677 21431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:36.465878 21431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:36.473253 21431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:36.473363 21431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:36.473567 21431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:24:36.484654 21431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:36.485960 21431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:36.487290 21431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:36.500401 21437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:36.500643 21440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:36.500800 21438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:36.502219 21431 server_base.cc:1034] running on GCE node
I20260809 20:24:36.502914 21431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:36.503844 21431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:36.505067 21431 hybrid_clock.cc:648] HybridClock initialized: now 1786307076505031 us; error 51 us; skew 500 ppm
I20260809 20:24:36.505374 21431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:36 dist-test-slave-rnrw krb5kdc[21319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307076, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:36.508458 21431 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:24:36.509944 21431 webserver.cc:460] Webserver started at http://127.12.78.1:35369/ using document root <none> and password file <none>
I20260809 20:24:36.510674 21431 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:36.510818 21431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:36.511133 21431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:36.514092 21431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "f42a475291ca433caa055a5d011de785"
format_stamp: "Formatted at 2026-08-09 20:24:36 on dist-test-slave-rnrw"
server_key: "16aca4ae46128e0511af223d6c8aba65"
server_key_iv: "dfb1dcb59c21273a11cda4bc5ffa1930"
server_key_version: "encryptionkey@0"
I20260809 20:24:36.514868 21431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "f42a475291ca433caa055a5d011de785"
format_stamp: "Formatted at 2026-08-09 20:24:36 on dist-test-slave-rnrw"
server_key: "16aca4ae46128e0511af223d6c8aba65"
server_key_iv: "dfb1dcb59c21273a11cda4bc5ffa1930"
server_key_version: "encryptionkey@0"
I20260809 20:24:36.520136 21431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 20:24:36.523694 21447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:36.525195 21431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:36.525324 21431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "f42a475291ca433caa055a5d011de785"
format_stamp: "Formatted at 2026-08-09 20:24:36 on dist-test-slave-rnrw"
server_key: "16aca4ae46128e0511af223d6c8aba65"
server_key_iv: "dfb1dcb59c21273a11cda4bc5ffa1930"
server_key_version: "encryptionkey@0"
I20260809 20:24:36.525487 21431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:36.551770 21431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:36.556320 21431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:36.556658 21431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:36.557605 21431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:36.558892 21431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:36.558966 21431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:36.559051 21431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:36.559095 21431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:36.591948 21431 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:38971
I20260809 20:24:36.592024 21565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:38971 every 8 connection(s)
I20260809 20:24:36.593298 21431 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:24:36.601089 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 21431
I20260809 20:24:36.601313 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:24:36.602072 12600 external_mini_cluster.cc:1428] Setting key 3c868e846c38a42f3b85081746a0904f
Aug 09 20:24:36 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307076, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:24:36.614986 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:36.595906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:40349 (local address 127.12.78.62:44301)
0809 20:24:36.596382 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:36.596387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:36.597368 (+ 981us) server_negotiation.cc:407] Connection header received
0809 20:24:36.599146 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:36.599151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:36.599217 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:36.599312 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:36.600805 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:36.601970 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:36.603944 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:36.604488 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:36.604656 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:36.608054 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:36.608064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:36.608072 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:36.608115 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:36.611727 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:36.612426 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:36.612434 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:36.612440 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:36.612627 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:36.613048 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:36.613056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:36.613062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:36.613445 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:36.613584 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:36.614552 (+ 968us) server_negotiation.cc:299] Negotiation successful
0809 20:24:36.614736 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":229,"threads_started":1}
I20260809 20:24:36.616582 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:36.596453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:40349)
0809 20:24:36.597206 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:36.597253 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:36.598909 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:36.599504 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:36.599512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:36.600029 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:36.600625 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:36.600647 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:36.602120 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:36.602123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:36.603795 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:36.603809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:36.604734 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:36.604739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:36.604926 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:36.605642 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:36.605664 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:36.607831 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:36.611872 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:36.611879 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:36.611891 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:36.612305 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:36.612751 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:36.612755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:36.612757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:36.612937 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:36.613648 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:36.613660 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:36.614106 (+ 446us) client_negotiation.cc:769] Sending connection context
0809 20:24:36.614465 (+ 359us) client_negotiation.cc:240] Negotiation successful
0809 20:24:36.615070 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":275,"threads_started":1}
I20260809 20:24:36.618186 21566 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44301
I20260809 20:24:36.618496 21566 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:36.619206 21566 heartbeater.cc:511] Master 127.12.78.62:44301 requested a full tablet report, sending...
I20260809 20:24:36.621335 21366 ts_manager.cc:194] Registered new tserver with Master: f42a475291ca433caa055a5d011de785 (127.12.78.1:38971)
I20260809 20:24:36.622757 21366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:40349
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:24:36.659509 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44301
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:36.990144 21574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:36.990491 21574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:36.990667 21574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:36.998004 21574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:36.998159 21574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:36.998346 21574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:24:37.008183 21574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:37.009588 21574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:37.010934 21574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:37.025094 21579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:37.025056 21580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:37.025133 21582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:37.025750 21574 server_base.cc:1034] running on GCE node
I20260809 20:24:37.026362 21574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:37.027287 21574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:37.028508 21574 hybrid_clock.cc:648] HybridClock initialized: now 1786307077028438 us; error 80 us; skew 500 ppm
I20260809 20:24:37.028914 21574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:37 dist-test-slave-rnrw krb5kdc[21319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307077, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:37.032259 21574 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:24:37.033859 21574 webserver.cc:460] Webserver started at http://127.12.78.2:40801/ using document root <none> and password file <none>
I20260809 20:24:37.034610 21574 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:37.034750 21574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:37.035076 21574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:37.038019 21574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "f4cbdf6f171c4821bfbbaee5c978887c"
format_stamp: "Formatted at 2026-08-09 20:24:37 on dist-test-slave-rnrw"
server_key: "46f94cd2c0b16d6282025b318e92c1c9"
server_key_iv: "5d58dabf4a4972c19af13c7ee7328b8f"
server_key_version: "encryptionkey@0"
I20260809 20:24:37.038792 21574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "f4cbdf6f171c4821bfbbaee5c978887c"
format_stamp: "Formatted at 2026-08-09 20:24:37 on dist-test-slave-rnrw"
server_key: "46f94cd2c0b16d6282025b318e92c1c9"
server_key_iv: "5d58dabf4a4972c19af13c7ee7328b8f"
server_key_version: "encryptionkey@0"
I20260809 20:24:37.043756 21574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 20:24:37.047125 21589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:37.048403 21574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 20:24:37.048615 21574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "f4cbdf6f171c4821bfbbaee5c978887c"
format_stamp: "Formatted at 2026-08-09 20:24:37 on dist-test-slave-rnrw"
server_key: "46f94cd2c0b16d6282025b318e92c1c9"
server_key_iv: "5d58dabf4a4972c19af13c7ee7328b8f"
server_key_version: "encryptionkey@0"
I20260809 20:24:37.048825 21574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:37.065615 21574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:37.069427 21574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:37.069720 21574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:37.070647 21574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:37.071965 21574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:37.072068 21574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:37.072223 21574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:37.072297 21574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:37.105309 21574 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:38087
I20260809 20:24:37.105365 21706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:38087 every 8 connection(s)
I20260809 20:24:37.106555 21574 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:24:37.113497 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 21574
I20260809 20:24:37.113695 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:24:37.114271 12600 external_mini_cluster.cc:1428] Setting key 6cd366f8ea9b4748a828711ba4b8ebe3
Aug 09 20:24:37 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307077, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:24:37.129200 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:37.109057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:53163 (local address 127.12.78.62:44301)
0809 20:24:37.109272 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:37.109278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:37.110441 (+ 1163us) server_negotiation.cc:407] Connection header received
0809 20:24:37.112145 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:37.112150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:37.112216 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:37.112304 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:37.113852 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:37.114893 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:37.116978 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:37.117447 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:37.117591 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:37.121841 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:37.121850 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:37.121857 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:37.121902 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:37.125656 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:37.126697 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:37.126701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:37.126703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:37.126828 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:37.127321 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:37.127324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:37.127326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:37.127597 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:37.127712 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:37.128691 (+ 979us) server_negotiation.cc:299] Negotiation successful
0809 20:24:37.128868 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 20:24:37.130784 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:37.109570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:53163)
0809 20:24:37.110274 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:37.110322 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:37.111895 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:37.112501 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:37.112508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:37.113065 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:37.113595 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:37.113639 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:37.115073 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:37.115077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:37.116813 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:37.116826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:37.117701 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:37.117712 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:37.118084 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:37.118835 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:37.118873 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:37.121593 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:37.125858 (+ 4265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:37.125870 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:37.125890 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:37.126558 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:37.126941 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:37.126947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:37.126951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:37.127209 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:37.127734 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:37.127742 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:37.128214 (+ 472us) client_negotiation.cc:769] Sending connection context
0809 20:24:37.128521 (+ 307us) client_negotiation.cc:240] Negotiation successful
0809 20:24:37.129165 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":284,"threads_started":1}
I20260809 20:24:37.132265 21707 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44301
I20260809 20:24:37.132622 21707 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:37.133407 21707 heartbeater.cc:511] Master 127.12.78.62:44301 requested a full tablet report, sending...
I20260809 20:24:37.135185 21366 ts_manager.cc:194] Registered new tserver with Master: f4cbdf6f171c4821bfbbaee5c978887c (127.12.78.2:38087)
I20260809 20:24:37.136447 21366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:53163
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:24:37.172835 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44301
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:37.506713 21712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:37.507084 21712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:37.507265 21712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:37.515434 21712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:37.515622 21712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:37.515827 21712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:24:37.526494 21712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:37.528002 21712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:37.529521 21712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:37.543486 21717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:37.543586 21721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:37.543536 21718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:37.544420 21712 server_base.cc:1034] running on GCE node
I20260809 20:24:37.545516 21712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:37.546682 21712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:37.547935 21712 hybrid_clock.cc:648] HybridClock initialized: now 1786307077547885 us; error 59 us; skew 500 ppm
I20260809 20:24:37.548331 21712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:37 dist-test-slave-rnrw krb5kdc[21319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307077, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:37.551918 21712 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:24:37.553723 21712 webserver.cc:460] Webserver started at http://127.12.78.3:37641/ using document root <none> and password file <none>
I20260809 20:24:37.554579 21712 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:37.554736 21712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:37.555056 21712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:37.558149 21712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "7d6f401ceb424530b8b92a524fb842f5"
format_stamp: "Formatted at 2026-08-09 20:24:37 on dist-test-slave-rnrw"
server_key: "712536458d3f76bd7d01747de646be08"
server_key_iv: "6a63e1208b43a8357daa5ce982d3d9f9"
server_key_version: "encryptionkey@0"
I20260809 20:24:37.558993 21712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "7d6f401ceb424530b8b92a524fb842f5"
format_stamp: "Formatted at 2026-08-09 20:24:37 on dist-test-slave-rnrw"
server_key: "712536458d3f76bd7d01747de646be08"
server_key_iv: "6a63e1208b43a8357daa5ce982d3d9f9"
server_key_version: "encryptionkey@0"
I20260809 20:24:37.564535 21712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260809 20:24:37.568229 21728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:37.569730 21712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 20:24:37.569926 21712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "7d6f401ceb424530b8b92a524fb842f5"
format_stamp: "Formatted at 2026-08-09 20:24:37 on dist-test-slave-rnrw"
server_key: "712536458d3f76bd7d01747de646be08"
server_key_iv: "6a63e1208b43a8357daa5ce982d3d9f9"
server_key_version: "encryptionkey@0"
I20260809 20:24:37.570152 21712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:37.586781 21712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:37.591030 21712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:37.591359 21712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:37.592430 21712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:37.593858 21712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:37.593966 21712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:37.594129 21712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:37.594206 21712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:37.625869 21566 heartbeater.cc:503] Master 127.12.78.62:44301 was elected leader, sending a full tablet report...
I20260809 20:24:37.628521 21712 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:40983
I20260809 20:24:37.628604 21844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:40983 every 8 connection(s)
I20260809 20:24:37.630084 21712 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:24:37.637836 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 21712
I20260809 20:24:37.638043 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:24:37.638593 12600 external_mini_cluster.cc:1428] Setting key 5b0f1c6fa7155c97572b5e57cc6c9422
Aug 09 20:24:37 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307077, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:24:37.650018 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:37.632646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:53155 (local address 127.12.78.62:44301)
0809 20:24:37.633087 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:37.633093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:37.634288 (+ 1195us) server_negotiation.cc:407] Connection header received
0809 20:24:37.636263 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:37.636268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:37.636348 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:37.636445 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:37.637959 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:37.639138 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:37.640937 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:37.641423 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:37.641538 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:37.644956 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:37.644961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:37.644964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:37.644997 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:37.647373 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:37.648070 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:37.648074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:37.648076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:37.648212 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:37.648653 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:37.648656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:37.648658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:37.648906 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:37.649017 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:37.649607 (+ 590us) server_negotiation.cc:299] Negotiation successful
0809 20:24:37.649764 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":214,"threads_started":1}
I20260809 20:24:37.650965 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:37.633254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:53155)
0809 20:24:37.634139 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:37.634181 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:37.636023 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:37.636723 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:37.636731 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:37.637217 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:37.637782 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:37.637800 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:37.639306 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:37.639311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:37.640794 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:37.640813 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:37.641695 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:37.641702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:37.641896 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:37.642600 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:37.642621 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:37.644764 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:37.647517 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:37.647552 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:37.647563 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:37.647954 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:37.648305 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:24:37.648309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:24:37.648311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:24:37.648476 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:24:37.649011 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:37.649016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:37.649298 (+ 282us) client_negotiation.cc:769] Sending connection context
0809 20:24:37.649509 (+ 211us) client_negotiation.cc:240] Negotiation successful
0809 20:24:37.649884 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":295,"threads_started":1}
I20260809 20:24:37.652421 21846 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44301
I20260809 20:24:37.652810 21846 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:37.653692 21846 heartbeater.cc:511] Master 127.12.78.62:44301 requested a full tablet report, sending...
I20260809 20:24:37.655179 21366 ts_manager.cc:194] Registered new tserver with Master: 7d6f401ceb424530b8b92a524fb842f5 (127.12.78.3:40983)
I20260809 20:24:37.655942 21366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:53155
I20260809 20:24:37.666370 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab.
Entry for principal oryx/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab.
WARNING: no policy specified for oryx/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab.
Entry for principal oryx/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab.
WARNING: no policy specified for oryx/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab.
Entry for principal oryx/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab.
I20260809 20:24:37.816337 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 21431
Aug 09 20:24:37 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:24:37.833714 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:37.817691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.12.78.1:38971)
0809 20:24:37.818401 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:37.818408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:37.818435 (+ 27us) server_negotiation.cc:407] Connection header received
0809 20:24:37.818497 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:37.818503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:37.818718 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:37.818866 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:37.820306 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:37.821408 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:37.822667 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:37.823221 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:37.823318 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:37.827463 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:37.827475 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:37.827489 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:37.827545 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:37.830828 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:37.831574 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:37.831581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:37.831583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:37.831734 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:37.832044 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:37.832053 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:37.832055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:37.832391 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:37.832692 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:37.832999 (+ 307us) server_negotiation.cc:299] Negotiation successful
0809 20:24:37.833268 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":400,"threads_started":1}
I20260809 20:24:37.835151 21561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:37.917667 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 21431
I20260809 20:24:37.936095 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 21574
Aug 09 20:24:37 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:24:37.951961 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:37.937109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.12.78.2:38087)
0809 20:24:37.937665 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:37.937672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:37.937698 (+ 26us) server_negotiation.cc:407] Connection header received
0809 20:24:37.937759 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:37.937765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:37.937972 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:37.938130 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:37.939437 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:37.940395 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:37.941733 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:37.942274 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:37.942369 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:37.946502 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:37.946512 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:37.946527 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:37.946564 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:37.949524 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:37.950148 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:37.950154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:37.950157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:37.950280 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:37.950600 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:37.950603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:37.950605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:37.950919 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:37.951174 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:37.951363 (+ 189us) server_negotiation.cc:299] Negotiation successful
0809 20:24:37.951556 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":281,"threads_started":1}
I20260809 20:24:37.953352 21701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:38.035420 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 21574
I20260809 20:24:38.053750 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 21712
Aug 09 20:24:38 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:24:38.070181 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:38.054852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44704 (local address 127.12.78.3:40983)
0809 20:24:38.055621 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:38.055627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:38.055655 (+ 28us) server_negotiation.cc:407] Connection header received
0809 20:24:38.055718 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:38.055724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:38.055943 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:38.056074 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:38.057569 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:38.058579 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:38.059902 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:38.060448 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:38.060676 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:38.065120 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:38.065131 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:38.065144 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:38.065174 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:38.067789 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:38.068378 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:38.068384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:38.068387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:38.068514 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:38.068885 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:38.068888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:38.068890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:38.069215 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:38.069452 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:38.069682 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 20:24:38.069870 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":464,"threads_started":1}
I20260809 20:24:38.071485 21840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:38.154989 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 21712
I20260809 20:24:38.172817 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 21333
I20260809 20:24:38.173997 21398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:38.278579 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 21333
I20260809 20:24:38.301775 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44301
--webserver_interface=127.12.78.62
--webserver_port=32961
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:44301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:24:38.636091 21865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:38.636379 21865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:38.636610 21865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:38.643808 21865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:38.643910 21865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:38.644013 21865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:38.644125 21865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:38.644202 21865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:24:38.654081 21865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:44301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=32961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:38.655400 21865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:38.656834 21865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:38.668816 21873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:38.668848 21871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:38.669070 21870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:38.669734 21865 server_base.cc:1034] running on GCE node
I20260809 20:24:38.670563 21865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:38.671908 21865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:38.673112 21865 hybrid_clock.cc:648] HybridClock initialized: now 1786307078673065 us; error 48 us; skew 500 ppm
I20260809 20:24:38.673425 21865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:38 dist-test-slave-rnrw krb5kdc[21319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307078, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:38.676627 21865 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:24:38.678076 21865 webserver.cc:460] Webserver started at http://127.12.78.62:32961/ using document root <none> and password file <none>
I20260809 20:24:38.678788 21865 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:38.678953 21865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:38.684309 21865 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 20:24:38.687253 21881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:38.688611 21865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 20:24:38.688788 21865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "f41b1f518635464da64584a571a02701"
format_stamp: "Formatted at 2026-08-09 20:24:35 on dist-test-slave-rnrw"
server_key: "23dd6d8cdef30fce7bb718045fad6e1c"
server_key_iv: "94a0f3c18c0846cb4d700f12ed12e3fa"
server_key_version: "encryptionkey@0"
I20260809 20:24:38.689736 21865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:38.701582 21865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:38.705392 21865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:38.705662 21865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:38.724184 21865 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:44301
I20260809 20:24:38.724206 21935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:44301 every 8 connection(s)
I20260809 20:24:38.725703 21865 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:24:38.732422 21936 sys_catalog.cc:263] Verifying existing consensus state
I20260809 20:24:38.735559 21936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701: Bootstrap starting.
I20260809 20:24:38.736078 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 21865
I20260809 20:24:38.736939 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:38971
--local_ip_for_outbound_sockets=127.12.78.1
--tserver_master_addrs=127.12.78.62:44301
--webserver_port=35369
--webserver_interface=127.12.78.1
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.78.1
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:24:38.755405 21936 log.cc:824] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:38.764201 21936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 20:24:38.764832 21936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701: Bootstrap complete.
I20260809 20:24:38.773811 21936 raft_consensus.cc:348] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41b1f518635464da64584a571a02701" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44301 } }
I20260809 20:24:38.774633 21936 raft_consensus.cc:729] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41b1f518635464da64584a571a02701, State: Initialized, Role: FOLLOWER
I20260809 20:24:38.775282 21936 consensus_queue.cc:260] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41b1f518635464da64584a571a02701" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44301 } }
I20260809 20:24:38.775534 21936 raft_consensus.cc:388] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:24:38.775669 21936 raft_consensus.cc:482] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:24:38.775856 21936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 1 FOLLOWER]: Advancing to term 2
I20260809 20:24:38.782317 21936 raft_consensus.cc:504] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41b1f518635464da64584a571a02701" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44301 } }
I20260809 20:24:38.782981 21936 leader_election.cc:302] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f41b1f518635464da64584a571a02701; no voters:
I20260809 20:24:38.783736 21936 leader_election.cc:288] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 20:24:38.784154 21940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 20:24:38.788205 21936 sys_catalog.cc:564] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:38.791811 21940 raft_consensus.cc:686] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [term 2 LEADER]: Becoming Leader. State: Replica: f41b1f518635464da64584a571a02701, State: Running, Role: LEADER
I20260809 20:24:38.792474 21940 consensus_queue.cc:237] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41b1f518635464da64584a571a02701" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44301 } }
I20260809 20:24:38.800279 21941 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f41b1f518635464da64584a571a02701" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41b1f518635464da64584a571a02701" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44301 } } }
I20260809 20:24:38.800731 21941 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:38.801301 21943 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f41b1f518635464da64584a571a02701. Latest consensus state: current_term: 2 leader_uuid: "f41b1f518635464da64584a571a02701" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41b1f518635464da64584a571a02701" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44301 } } }
I20260809 20:24:38.801632 21943 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:38.804471 21952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:38.808990 21952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:38.810019 21952 catalog_manager.cc:1259] Loaded cluster ID: d82a35e28e6a4a1dab3951efb2ef92ba
I20260809 20:24:38.810127 21952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:38.812780 21952 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:38.813855 21952 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f41b1f518635464da64584a571a02701: Loaded TSK: 0
I20260809 20:24:38.815021 21952 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 20:24:39.095695 21938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:39.096009 21938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:39.096175 21938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:39.103590 21938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:39.103744 21938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:39.103914 21938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:24:39.113790 21938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.1.keytab
--principal=oryx/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:38971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=35369
--tserver_master_addrs=127.12.78.62:44301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:39.115111 21938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:39.116385 21938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:39.131181 21965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:39.131217 21966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:39.131465 21968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:39.131783 21938 server_base.cc:1034] running on GCE node
I20260809 20:24:39.132381 21938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:39.133343 21938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:39.134572 21938 hybrid_clock.cc:648] HybridClock initialized: now 1786307079134526 us; error 41 us; skew 500 ppm
I20260809 20:24:39.134884 21938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307079, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:39.138060 21938 init.cc:377] Logged in from keytab as oryx/127.12.78.1@KRBTEST.COM (short username oryx)
I20260809 20:24:39.139599 21938 webserver.cc:460] Webserver started at http://127.12.78.1:35369/ using document root <none> and password file <none>
I20260809 20:24:39.140267 21938 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:39.140419 21938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:39.145697 21938 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:24:39.148489 21975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:39.149922 21938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 20:24:39.150084 21938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "f42a475291ca433caa055a5d011de785"
format_stamp: "Formatted at 2026-08-09 20:24:36 on dist-test-slave-rnrw"
server_key: "16aca4ae46128e0511af223d6c8aba65"
server_key_iv: "dfb1dcb59c21273a11cda4bc5ffa1930"
server_key_version: "encryptionkey@0"
I20260809 20:24:39.151031 21938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:39.170660 21938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:39.175053 21938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:39.175343 21938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:39.176365 21938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:39.177814 21938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:39.177909 21938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:39.178069 21938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:39.178149 21938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:39.212770 21938 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:38971
I20260809 20:24:39.212854 22091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:38971 every 8 connection(s)
I20260809 20:24:39.214210 21938 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:24:39.222921 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 21938
I20260809 20:24:39.223752 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:38087
--local_ip_for_outbound_sockets=127.12.78.2
--tserver_master_addrs=127.12.78.62:44301
--webserver_port=40801
--webserver_interface=127.12.78.2
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.78.2
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:39.233301 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:39.217152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:60363)
0809 20:24:39.217957 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:39.218001 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:39.219714 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:39.221121 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:39.221128 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:39.221608 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:39.222131 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:39.222148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.224741 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.224746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:39.225929 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:39.225941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.227507 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.227512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:39.227702 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:39.228382 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:39.228409 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:39.232004 (+ 3595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":340,"threads_started":1}
W20260809 20:24:39.233386 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:39.217155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:60363 (local address 127.12.78.62:44301)
0809 20:24:39.217870 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:39.217882 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:39.218122 (+ 240us) server_negotiation.cc:407] Connection header received
0809 20:24:39.220230 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:39.220243 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:39.220665 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:39.220972 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:39.222333 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.224571 (+ 2238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.226092 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.227274 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.227509 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:39.232482 (+ 4973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:60363: BlockingRecv error: recv got EOF from 127.12.78.1:60363 (error 108)
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":285,"threads_started":1}
W20260809 20:24:39.234411 22092 heartbeater.cc:644] Failed to heartbeat to 127.12.78.62:44301 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.78.62:44301: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 20:24:39.244249 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:39.235151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:49639 (local address 127.12.78.62:44301)
0809 20:24:39.235352 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:39.235358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:39.235378 (+ 20us) server_negotiation.cc:407] Connection header received
0809 20:24:39.235480 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:39.235486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:39.235559 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:39.235667 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:39.236351 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.237206 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.238502 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.238941 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.239094 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:39.244073 (+ 4979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:49639: BlockingRecv error: recv got EOF from 127.12.78.1:49639 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 20:24:39.244773 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:39.235060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:49639)
0809 20:24:39.235243 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:39.235266 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:39.235373 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:39.235693 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:39.235698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:39.236066 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:39.236203 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:39.236211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.237359 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.237365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:39.238336 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:39.238347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.239110 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.239116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:39.239323 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:39.239979 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:39.239997 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:39.244490 (+ 4493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:39.253700 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:39.245460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:53095)
0809 20:24:39.245620 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:39.245647 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:39.245755 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:39.246155 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:39.246160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:39.246518 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:39.246672 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:39.246681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.247776 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.247782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:39.248843 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:39.248855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.249616 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.249622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:39.249830 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:39.250445 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:39.250463 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:39.253362 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260809 20:24:39.253886 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:39.245666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:53095 (local address 127.12.78.62:44301)
0809 20:24:39.245866 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:39.245872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:39.245888 (+ 16us) server_negotiation.cc:407] Connection header received
0809 20:24:39.245940 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:39.245944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:39.246012 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:39.246128 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:39.246822 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.247626 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.248999 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.249448 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.249603 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:39.253565 (+ 3962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:53095: BlockingRecv error: recv got EOF from 127.12.78.1:53095 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260809 20:24:39.254002 22092 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 20:24:39.562780 22096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:39.563053 22096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:39.563148 22096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:39.570272 22096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:39.570384 22096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:39.570503 22096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:24:39.580114 22096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.2.keytab
--principal=oryx/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:38087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=40801
--tserver_master_addrs=127.12.78.62:44301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:39.581406 22096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:39.582697 22096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:39.595659 22101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:39.595866 22102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:39.596354 22104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:39.596462 22096 server_base.cc:1034] running on GCE node
I20260809 20:24:39.597113 22096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:39.597970 22096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:39.599176 22096 hybrid_clock.cc:648] HybridClock initialized: now 1786307079599110 us; error 60 us; skew 500 ppm
I20260809 20:24:39.599510 22096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307079, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:39.602695 22096 init.cc:377] Logged in from keytab as oryx/127.12.78.2@KRBTEST.COM (short username oryx)
I20260809 20:24:39.604107 22096 webserver.cc:460] Webserver started at http://127.12.78.2:40801/ using document root <none> and password file <none>
I20260809 20:24:39.604851 22096 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:39.605017 22096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:39.610275 22096 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 20:24:39.612907 22111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:39.614208 22096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 20:24:39.614382 22096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "f4cbdf6f171c4821bfbbaee5c978887c"
format_stamp: "Formatted at 2026-08-09 20:24:37 on dist-test-slave-rnrw"
server_key: "46f94cd2c0b16d6282025b318e92c1c9"
server_key_iv: "5d58dabf4a4972c19af13c7ee7328b8f"
server_key_version: "encryptionkey@0"
I20260809 20:24:39.615321 22096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:39.634558 22096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:39.638445 22096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:39.638724 22096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:39.639727 22096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:39.641093 22096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:39.641249 22096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:39.641388 22096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:39.641484 22096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:39.676738 22096 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:38087
I20260809 20:24:39.676798 22229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:38087 every 8 connection(s)
I20260809 20:24:39.678158 22096 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:24:39.679970 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 22096
I20260809 20:24:39.680696 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:40983
--local_ip_for_outbound_sockets=127.12.78.3
--tserver_master_addrs=127.12.78.62:44301
--webserver_port=37641
--webserver_interface=127.12.78.3
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.78.3
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 20:24:39.695268 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:39.680862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:35093 (local address 127.12.78.62:44301)
0809 20:24:39.681069 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:39.681074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:39.682426 (+ 1352us) server_negotiation.cc:407] Connection header received
0809 20:24:39.684275 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:39.684284 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:39.684378 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:39.684509 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:39.686043 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.687190 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.688960 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.689508 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.689649 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:39.694999 (+ 5350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:35093: BlockingRecv error: recv got EOF from 127.12.78.2:35093 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 20:24:39.695907 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:39.681395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:35093)
0809 20:24:39.682232 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:39.682277 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:39.683995 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:39.684797 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:39.684805 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:39.685302 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:39.685844 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:39.685862 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.687360 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.687364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:39.688773 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:39.688804 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.689745 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.689754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:39.690060 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:39.690740 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:39.690761 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:39.694805 (+ 4044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":355,"threads_started":1}
W20260809 20:24:39.696715 22230 heartbeater.cc:644] Failed to heartbeat to 127.12.78.62:44301 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.78.62:44301: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:39.705024 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:39.697310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:58991)
0809 20:24:39.697492 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:39.697511 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:39.697627 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:39.697947 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:39.697952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:39.698278 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:39.698405 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:39.698413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.699373 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.699379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:39.700353 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:39.700365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.701135 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.701141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:39.701315 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:39.701911 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:39.701928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:39.704825 (+ 2897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 20:24:39.705662 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:39.697395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:58991 (local address 127.12.78.62:44301)
0809 20:24:39.697560 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:39.697565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:39.697584 (+ 19us) server_negotiation.cc:407] Connection header received
0809 20:24:39.697748 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:39.697754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:39.697824 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:39.697924 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:39.698546 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.699234 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.700496 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.700959 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.701148 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:39.705363 (+ 4215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:58991: BlockingRecv error: recv got EOF from 127.12.78.2:58991 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:39 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:39.713963 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:39.705703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:34461)
0809 20:24:39.706143 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:39.706164 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:39.706308 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:39.706625 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:39.706630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:39.706931 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:39.707065 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:39.707073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.708015 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.708021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:39.708971 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:39.708983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.709881 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.709885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:39.710041 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:39.710656 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:39.710673 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:39.713662 (+ 2989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301}
W20260809 20:24:39.714126 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:39.705792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:34461 (local address 127.12.78.62:44301)
0809 20:24:39.705997 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:39.706005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:39.706228 (+ 223us) server_negotiation.cc:407] Connection header received
0809 20:24:39.706424 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:39.706430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:39.706496 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:39.706595 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:39.707202 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.707878 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.709122 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:39.709601 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:39.709726 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:39.713871 (+ 4145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:34461: BlockingRecv error: recv got EOF from 127.12.78.2:34461 (error 108)
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":345,"threads_started":1}
W20260809 20:24:39.714370 22230 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 20:24:40.024346 22233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:40.024732 22233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:40.024912 22233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:40.032181 22233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:40.032347 22233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:40.032572 22233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:24:40.042459 22233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/krb5kdc/oryx_127.12.78.3.keytab
--principal=oryx/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:40983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=37641
--tserver_master_addrs=127.12.78.62:44301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:40.043825 22233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:40.045212 22233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:40.058218 22239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:40.058151 22242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:40.058197 22240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:40.059180 22233 server_base.cc:1034] running on GCE node
I20260809 20:24:40.059803 22233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:40.060823 22233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:40.062031 22233 hybrid_clock.cc:648] HybridClock initialized: now 1786307080061987 us; error 46 us; skew 500 ppm
I20260809 20:24:40.062355 22233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:24:40 dist-test-slave-rnrw krb5kdc[21319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307080, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:24:40.065883 22233 init.cc:377] Logged in from keytab as oryx/127.12.78.3@KRBTEST.COM (short username oryx)
I20260809 20:24:40.067566 22233 webserver.cc:460] Webserver started at http://127.12.78.3:37641/ using document root <none> and password file <none>
I20260809 20:24:40.068228 22233 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:40.068344 22233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:40.073875 22233 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260809 20:24:40.076435 22250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:40.077960 22233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:40.078169 22233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "7d6f401ceb424530b8b92a524fb842f5"
format_stamp: "Formatted at 2026-08-09 20:24:37 on dist-test-slave-rnrw"
server_key: "712536458d3f76bd7d01747de646be08"
server_key_iv: "6a63e1208b43a8357daa5ce982d3d9f9"
server_key_version: "encryptionkey@0"
I20260809 20:24:40.079277 22233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:40.096640 22233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:40.101670 22233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:40.101989 22233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:40.102998 22233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:40.104341 22233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:40.104432 22233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:40.104619 22233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:40.104718 22233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:40.138585 22233 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:40983
I20260809 20:24:40.138679 22367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:40983 every 8 connection(s)
I20260809 20:24:40.140098 22233 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:24:40.148002 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 22233
Aug 09 20:24:40 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:40 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 20:24:40.159600 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:40.142808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:46859 (local address 127.12.78.62:44301)
0809 20:24:40.143007 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:40.143012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:40.144173 (+ 1161us) server_negotiation.cc:407] Connection header received
0809 20:24:40.146194 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:40.146200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:40.146308 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:40.146432 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:40.148354 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.149256 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.151810 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.152476 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.152654 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:40.159307 (+ 6653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:46859: BlockingRecv error: recv got EOF from 127.12.78.3:46859 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 20:24:40.160369 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:40.143205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:46859)
0809 20:24:40.143979 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:40.144030 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:40.145886 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:40.146699 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:40.146707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:40.147253 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:40.148139 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:40.148170 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.149484 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.149492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:40.151645 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:40.151663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.152731 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.152736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:40.152939 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:40.153747 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:40.153780 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:40.159097 (+ 5317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":310,"threads_started":1}
W20260809 20:24:40.161309 22368 heartbeater.cc:644] Failed to heartbeat to 127.12.78.62:44301 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.78.62:44301: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
I20260809 20:24:40.168277 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:40.149233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.12.78.62:44301)
0809 20:24:40.149501 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:40.149507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:40.149909 (+ 402us) server_negotiation.cc:407] Connection header received
0809 20:24:40.150107 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:40.150111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:40.150170 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:40.150264 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:40.152190 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.153371 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.155384 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.156119 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.156257 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:40.159767 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:40.159782 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:40.159808 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:40.159860 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:40.164343 (+ 4483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:40.165251 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:40.165259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:40.165267 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:40.165524 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:24:40.165945 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:24:40.165949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:24:40.165951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:24:40.166712 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:40.167082 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:40.167518 (+ 436us) server_negotiation.cc:299] Negotiation successful
0809 20:24:40.167989 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Aug 09 20:24:40 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:40 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:40.171378 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:40.161963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:44591)
0809 20:24:40.162149 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:40.162168 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:40.162259 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:40.162704 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:40.162707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:40.163005 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:40.163146 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:40.163154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.164712 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.164718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:40.165999 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:40.166011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.167003 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.167009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:40.167191 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:40.167839 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:40.167866 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:40.171079 (+ 3213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 20:24:40.171506 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:40.162162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:44591 (local address 127.12.78.62:44301)
0809 20:24:40.162383 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:40.162389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:40.162409 (+ 20us) server_negotiation.cc:407] Connection header received
0809 20:24:40.162465 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:40.162473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:40.162561 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:40.162697 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:40.163293 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.164509 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.166133 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.166858 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.167012 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:40.171273 (+ 4261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:44591: BlockingRecv error: recv got EOF from 127.12.78.3:44591 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 20:24:40 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:40 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:40.180013 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:40.172222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:53887)
0809 20:24:40.172406 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:40.172428 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:40.172652 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:40.172960 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:40.172965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:40.173252 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:40.173403 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:40.173411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.174377 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.174380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:40.175410 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:40.175422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.176061 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.176066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:40.176231 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:40.176914 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:40.176932 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:40.179793 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 20:24:40.180346 22368 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 20:24:40.180440 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:40.172321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:53887 (local address 127.12.78.62:44301)
0809 20:24:40.172537 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:40.172584 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:40.172605 (+ 21us) server_negotiation.cc:407] Connection header received
0809 20:24:40.172771 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:40.172776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:40.172839 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:40.172915 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:40.173569 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.174257 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.175544 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.175933 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.176021 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:40.180004 (+ 3983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:53887: BlockingRecv error: recv got EOF from 127.12.78.3:53887 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Aug 09 20:24:40 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:40 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:40.264441 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:40.254789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:48253)
0809 20:24:40.255377 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:40.255393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:40.255515 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:40.255939 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:40.255943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:40.256623 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:40.256826 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:40.256836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.258306 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.258311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:40.259439 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:40.259454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.260178 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.260183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:40.260316 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:40.261004 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:40.261034 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:40.264206 (+ 3172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":342,"threads_started":1}
W20260809 20:24:40.264707 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:40.254827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:48253 (local address 127.12.78.62:44301)
0809 20:24:40.255009 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:40.255014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:40.255490 (+ 476us) server_negotiation.cc:407] Connection header received
0809 20:24:40.255696 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:40.255702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:40.255781 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:40.255908 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:40.256974 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.258117 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.259588 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.259995 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.260096 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:40.264455 (+ 4359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:48253: BlockingRecv error: recv got EOF from 127.12.78.1:48253 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 20:24:40 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:40 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:40.723946 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:40.714921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:37547)
0809 20:24:40.715446 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:40.715461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:40.715634 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:40.715964 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:40.715969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:40.716589 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:40.716867 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:40.716878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.717887 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.717893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:40.718942 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:40.718954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.719747 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.719752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:40.719869 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:40.720508 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:40.720526 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:40.723614 (+ 3088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":290,"threads_started":1}
W20260809 20:24:40.724299 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:40.715001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:37547 (local address 127.12.78.62:44301)
0809 20:24:40.715237 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:40.715246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:40.715545 (+ 299us) server_negotiation.cc:407] Connection header received
0809 20:24:40.715767 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:40.715771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:40.715832 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:40.715934 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:40.717007 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.717736 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.719148 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:40.719586 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:40.719681 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:40.723846 (+ 4165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:37547: BlockingRecv error: recv got EOF from 127.12.78.2:37547 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 20:24:41 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:41 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:41.195626 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:41.181090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:56099)
0809 20:24:41.181612 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:41.181634 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:41.181802 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:41.182183 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:41.182189 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:41.182752 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:41.182998 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:41.183009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:41.184237 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:41.184245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:41.185507 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:41.185522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:41.186395 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:41.186402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:41.186617 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:41.192002 (+ 5385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:41.192076 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:41.195389 (+ 3313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":243,"threads_started":1}
W20260809 20:24:41.195889 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:41.181126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:56099 (local address 127.12.78.62:44301)
0809 20:24:41.181360 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:41.181369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:41.181730 (+ 361us) server_negotiation.cc:407] Connection header received
0809 20:24:41.181948 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:41.181954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:41.182034 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:41.182149 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:41.183169 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:41.184071 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:41.185681 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:41.186173 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:41.186337 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:41.195601 (+ 9264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:56099: BlockingRecv error: recv got EOF from 127.12.78.3:56099 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 20:24:41 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:41 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:41.274020 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:41.265454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:49667)
0809 20:24:41.266103 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:41.266130 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:41.266250 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:41.266668 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:41.266675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:41.267102 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:41.267251 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:41.267261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:41.268317 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:41.268321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:41.269273 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:41.269285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:41.269965 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:41.269969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:41.270104 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:41.270811 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:41.270830 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:41.273764 (+ 2934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":309,"threads_started":1}
W20260809 20:24:41.274222 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:41.265539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:49667 (local address 127.12.78.62:44301)
0809 20:24:41.265771 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:41.265780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:41.266221 (+ 441us) server_negotiation.cc:407] Connection header received
0809 20:24:41.266390 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:41.266394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:41.266458 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:41.266581 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:41.267384 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:41.268184 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:41.269402 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:41.269819 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:41.269917 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:41.273968 (+ 4051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:49667: BlockingRecv error: recv got EOF from 127.12.78.1:49667 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 09 20:24:41 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:41 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:41.733822 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:41.724970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:57579)
0809 20:24:41.725496 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:41.725514 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:41.725699 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:41.726085 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:41.726090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:41.726618 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:41.726828 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:41.726839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:41.727933 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:41.727942 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:41.729238 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:41.729250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:41.730011 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:41.730018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:41.730180 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:41.730802 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:41.730821 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:41.733586 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":275,"threads_started":1}
W20260809 20:24:41.734038 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:41.725098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:57579 (local address 127.12.78.62:44301)
0809 20:24:41.725351 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:41.725357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:41.725616 (+ 259us) server_negotiation.cc:407] Connection header received
0809 20:24:41.725862 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:41.725868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:41.725939 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:41.726048 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:41.726975 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:41.727762 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:41.729384 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:41.729834 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:41.729966 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:41.733773 (+ 3807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:57579: BlockingRecv error: recv got EOF from 127.12.78.2:57579 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Aug 09 20:24:42 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:42 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:42.204717 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:42.196617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:34917)
0809 20:24:42.197166 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:42.197183 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:42.197296 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:42.197567 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:42.197570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:42.197954 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:42.198111 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:42.198120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:42.199116 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:42.199121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:42.200087 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:42.200099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:42.200816 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:42.200820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:42.200931 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:42.201548 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:42.201566 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:42.204349 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":291,"threads_started":1}
W20260809 20:24:42.204945 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:42.196842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:34917 (local address 127.12.78.62:44301)
0809 20:24:42.197016 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:42.197020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:42.197256 (+ 236us) server_negotiation.cc:407] Connection header received
0809 20:24:42.197395 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:42.197399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:42.197471 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:42.197552 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:42.198243 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:42.198960 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:42.200224 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:42.200681 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:42.200771 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:42.204648 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:34917: BlockingRecv error: recv got EOF from 127.12.78.3:34917 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 20:24:42 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:42 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:42.284047 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:42.275018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:56219)
0809 20:24:42.275588 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:42.275604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:42.275767 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:42.276142 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:42.276147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:42.276535 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:42.276761 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:42.276769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:42.277770 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:42.277779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:42.278715 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:42.278727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:42.279411 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:42.279416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:42.279655 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:42.280400 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:42.280424 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:42.283680 (+ 3256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":304,"threads_started":1}
W20260809 20:24:42.284183 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:42.275143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:56219 (local address 127.12.78.62:44301)
0809 20:24:42.275379 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:42.275386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:42.275692 (+ 306us) server_negotiation.cc:407] Connection header received
0809 20:24:42.275895 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:42.275906 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:42.275970 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:42.276105 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:42.276901 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:42.277630 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:42.278857 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:42.279263 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:42.279398 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:42.283882 (+ 4484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:56219: BlockingRecv error: recv got EOF from 127.12.78.1:56219 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 09 20:24:42 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:42 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:42.742803 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:42.734727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:47423)
0809 20:24:42.735266 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:42.735282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:42.735394 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:42.735707 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:42.735711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:42.736033 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:42.736225 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:42.736235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:42.737226 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:42.737230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:42.738143 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:42.738155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:42.738795 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:42.738799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:42.738928 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:42.739578 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:42.739595 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:42.742441 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":317,"threads_started":1}
W20260809 20:24:42.742939 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:42.734760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:47423 (local address 127.12.78.62:44301)
0809 20:24:42.734987 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:42.734995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:42.735360 (+ 365us) server_negotiation.cc:407] Connection header received
0809 20:24:42.735489 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:42.735493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:42.735585 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:42.735665 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:42.736341 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:42.737111 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:42.738272 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:42.738676 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:42.738767 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:42.742691 (+ 3924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:47423: BlockingRecv error: recv got EOF from 127.12.78.2:47423 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 20:24:43 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:43 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:43.214643 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:43.205635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:46569)
0809 20:24:43.206160 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:43.206175 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:43.206294 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:43.206661 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:43.206664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:43.207002 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:43.207194 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:43.207202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:43.208274 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:43.208278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:43.209254 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:43.209266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:43.210187 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:43.210193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:43.210416 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:43.211172 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:43.211201 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:43.214314 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":292,"threads_started":1}
W20260809 20:24:43.214807 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:43.205797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:46569 (local address 127.12.78.62:44301)
0809 20:24:43.205990 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:43.205994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:43.206268 (+ 274us) server_negotiation.cc:407] Connection header received
0809 20:24:43.206404 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:43.206408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:43.206515 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:43.206623 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:43.207337 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:43.208157 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:43.209403 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:43.210016 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:43.210190 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:43.214550 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:46569: BlockingRecv error: recv got EOF from 127.12.78.3:46569 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 20:24:43 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:43 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:43.294056 22383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:43.285043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:48723)
0809 20:24:43.285642 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:43.285660 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:43.285788 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:43.286091 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:43.286094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:43.286511 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:43.286680 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:43.286689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:43.287660 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:43.287663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:43.288839 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:43.288851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:43.289592 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:43.289596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:43.289780 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:43.290377 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:43.290396 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:43.293626 (+ 3230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":338,"threads_started":1}
W20260809 20:24:43.294162 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:43.285100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:48723 (local address 127.12.78.62:44301)
0809 20:24:43.285280 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:43.285284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:43.285737 (+ 453us) server_negotiation.cc:407] Connection header received
0809 20:24:43.285888 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:43.285892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:43.285951 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:43.286032 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:43.286803 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:43.287533 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:43.288974 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:43.289464 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:43.289565 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:43.293876 (+ 4311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:48723: BlockingRecv error: recv got EOF from 127.12.78.1:48723 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 09 20:24:43 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:43 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:43.752090 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:43.743885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:38431)
0809 20:24:43.744417 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:43.744433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:43.744606 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:43.744954 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:43.744957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:43.745405 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:43.745561 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:43.745570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:43.746547 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:43.746551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:43.747510 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:43.747521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:43.748222 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:43.748226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:43.748348 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:43.748964 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:43.748981 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:43.751864 (+ 2883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":301,"threads_started":1}
W20260809 20:24:43.752326 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:43.743976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:38431 (local address 127.12.78.62:44301)
0809 20:24:43.744167 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:43.744171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:43.744574 (+ 403us) server_negotiation.cc:407] Connection header received
0809 20:24:43.744744 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:43.744748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:43.744822 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:43.744909 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:43.745716 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:43.746426 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:43.747646 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:43.748064 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:43.748173 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:43.752066 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:38431: BlockingRecv error: recv got EOF from 127.12.78.2:38431 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 09 20:24:44 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:44 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:44.223835 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:44.215521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:46345)
0809 20:24:44.216060 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:44.216077 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:44.216187 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:44.216487 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:44.216491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:44.216891 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:44.217047 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:44.217057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:44.218024 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:44.218029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:44.219051 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:44.219064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:44.219785 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:44.219789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:44.219928 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:44.220607 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:44.220625 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:44.223492 (+ 2867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":298,"threads_started":1}
W20260809 20:24:44.223953 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:44.215660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:46345 (local address 127.12.78.62:44301)
0809 20:24:44.215842 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:44.215848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:44.216153 (+ 305us) server_negotiation.cc:407] Connection header received
0809 20:24:44.216283 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:44.216287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:44.216358 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:44.216440 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:44.217179 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:44.217905 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:44.219177 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:44.219647 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:44.219774 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:44.223698 (+ 3924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:46345: BlockingRecv error: recv got EOF from 127.12.78.3:46345 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 20:24:44 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:44 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:44.303786 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:44.295026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:58483)
0809 20:24:44.295608 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:44.295625 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:44.295787 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:44.296185 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:44.296190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:44.296820 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:44.297030 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:44.297041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:44.298064 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:44.298071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:44.298991 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:44.299004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:44.299721 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:44.299726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:44.299910 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:44.300663 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:44.300681 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:44.303549 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":258,"threads_started":1}
W20260809 20:24:44.303962 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:44.295096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:58483 (local address 127.12.78.62:44301)
0809 20:24:44.295339 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:44.295345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:44.295754 (+ 409us) server_negotiation.cc:407] Connection header received
0809 20:24:44.295941 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:44.295950 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:44.296012 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:44.296141 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:44.297194 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:44.297919 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:44.299143 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:44.299569 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:44.299698 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:44.303738 (+ 4040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:58483: BlockingRecv error: recv got EOF from 127.12.78.1:58483 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 20:24:44 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:44 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:44.761611 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:44.752999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:57463)
0809 20:24:44.753559 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:44.753576 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:44.753688 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:44.754014 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:44.754019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:44.754525 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:44.754687 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:44.754696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:44.755785 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:44.755789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:44.756796 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:44.756808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:44.757501 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:44.757507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:44.757703 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:44.758274 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:44.758292 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:44.761254 (+ 2962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":283,"threads_started":1}
W20260809 20:24:44.761797 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:44.753177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:57463 (local address 127.12.78.62:44301)
0809 20:24:44.753358 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:44.753363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:44.753649 (+ 286us) server_negotiation.cc:407] Connection header received
0809 20:24:44.753788 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:44.753792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:44.753851 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:44.753933 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:44.754853 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:44.755643 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:44.756919 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:44.757346 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:44.757437 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:44.761489 (+ 4052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:57463: BlockingRecv error: recv got EOF from 127.12.78.2:57463 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 20:24:45 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:45 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:45.233695 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:45.225027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:50029)
0809 20:24:45.225621 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:45.225638 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:45.225793 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:45.226163 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:45.226169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:45.226651 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:45.226818 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:45.226828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:45.227852 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:45.227858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:45.228793 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:45.228805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:45.229521 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:45.229527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:45.229711 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:45.230419 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:45.230436 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:45.233452 (+ 3016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":312,"threads_started":1}
W20260809 20:24:45.233904 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:45.225140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:50029 (local address 127.12.78.62:44301)
0809 20:24:45.225344 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:45.225355 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:45.225735 (+ 380us) server_negotiation.cc:407] Connection header received
0809 20:24:45.225927 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:45.225932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:45.226014 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:45.226122 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:45.226973 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:45.227701 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:45.228939 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:45.229375 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:45.229506 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:45.233649 (+ 4143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:50029: BlockingRecv error: recv got EOF from 127.12.78.3:50029 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 20:24:45 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:45 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:45.313308 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:45.304745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:51955)
0809 20:24:45.305287 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:45.305303 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:45.305464 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:45.305806 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:45.305810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:45.306216 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:45.306367 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:45.306375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:45.307429 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:45.307448 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:45.308369 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:45.308381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:45.309218 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:45.309225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:45.309414 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:45.310135 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:45.310153 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:45.313078 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":324,"threads_started":1}
W20260809 20:24:45.313545 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:45.304810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:51955 (local address 127.12.78.62:44301)
0809 20:24:45.305034 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:45.305042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:45.305398 (+ 356us) server_negotiation.cc:407] Connection header received
0809 20:24:45.305591 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:45.305602 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:45.305664 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:45.305769 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:45.306518 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:45.307283 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:45.308529 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:45.309040 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:45.309222 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:45.313308 (+ 4086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:51955: BlockingRecv error: recv got EOF from 127.12.78.1:51955 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 09 20:24:45 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:45 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:45.770818 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:45.762520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:40863)
0809 20:24:45.763036 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:45.763053 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:45.763171 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:45.763467 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:45.763470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:45.763857 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:45.764014 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:45.764022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:45.765027 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:45.765031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:45.765936 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:45.765948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:45.766719 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:45.766723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:45.766843 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:45.767409 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:45.767426 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:45.770602 (+ 3176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":299,"threads_started":1}
W20260809 20:24:45.771107 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:45.762615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:40863 (local address 127.12.78.62:44301)
0809 20:24:45.762875 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:45.762882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:45.763137 (+ 255us) server_negotiation.cc:407] Connection header received
0809 20:24:45.763275 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:45.763278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:45.763340 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:45.763423 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:45.764157 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:45.764902 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:45.766091 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:45.766572 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:45.766677 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:45.770828 (+ 4151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:40863: BlockingRecv error: recv got EOF from 127.12.78.2:40863 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 09 20:24:46 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:46 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:46.247745 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:46.234902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:43669)
0809 20:24:46.235539 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:46.235556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:46.235723 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:46.236078 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:46.236083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:46.236494 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:46.236705 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:46.236714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:46.237740 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:46.237746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:46.238665 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:46.238676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:46.239468 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:46.239474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:46.239667 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:46.243534 (+ 3867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:46.243560 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:46.247380 (+ 3820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":326,"threads_started":1}
W20260809 20:24:46.248010 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:46.235259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:43669 (local address 127.12.78.62:44301)
0809 20:24:46.235460 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:46.235467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:46.235645 (+ 178us) server_negotiation.cc:407] Connection header received
0809 20:24:46.235853 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:46.235859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:46.235934 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:46.236038 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:46.236850 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:46.237598 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:46.238807 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:46.239300 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:46.239447 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:46.247704 (+ 8257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:43669: BlockingRecv error: recv got EOF from 127.12.78.3:43669 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 20:24:46 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:46 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:46.323376 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:46.314292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:35387)
0809 20:24:46.314948 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:46.314965 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:46.315084 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:46.315387 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:46.315391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:46.315885 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:46.316061 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:46.316070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:46.317106 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:46.317110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:46.318051 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:46.318063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:46.318878 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:46.318883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:46.319026 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:46.319834 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:46.319853 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:46.323009 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":318,"threads_started":1}
W20260809 20:24:46.323480 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:46.314419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:35387 (local address 127.12.78.62:44301)
0809 20:24:46.314590 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:46.314595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:46.315042 (+ 447us) server_negotiation.cc:407] Connection header received
0809 20:24:46.315184 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:46.315188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:46.315251 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:46.315339 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:46.316194 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:46.316962 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:46.318226 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:46.318722 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:46.318826 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:46.323249 (+ 4423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:35387: BlockingRecv error: recv got EOF from 127.12.78.1:35387 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 20:24:46 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:46 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:46.782924 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:46.771755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:58571)
0809 20:24:46.772384 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:46.772402 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:46.772593 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:46.772964 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:46.772969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:46.773465 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:46.773653 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:46.773664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:46.774769 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:46.774777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:46.775814 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:46.775827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:46.776929 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:46.776936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:46.777193 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:46.778038 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:46.778063 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:46.782523 (+ 4460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":306,"threads_started":1}
W20260809 20:24:46.783102 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:46.771866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:58571 (local address 127.12.78.62:44301)
0809 20:24:46.772106 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:46.772116 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:46.772502 (+ 386us) server_negotiation.cc:407] Connection header received
0809 20:24:46.772754 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:46.772758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:46.772821 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:46.772933 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:46.773820 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:46.774594 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:46.775969 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:46.776699 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:46.776840 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:46.782777 (+ 5937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:58571: BlockingRecv error: recv got EOF from 127.12.78.2:58571 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 09 20:24:47 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:47 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:47.258685 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:47.248938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:46469)
0809 20:24:47.249737 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:47.249754 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:47.249894 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:47.250251 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:47.250257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:47.250938 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:47.251198 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:47.251214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:47.252598 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:47.252603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:47.253512 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:47.253554 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:47.254363 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:47.254371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:47.254597 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:47.255208 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:47.255228 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:47.258421 (+ 3193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":329,"threads_started":1}
W20260809 20:24:47.258823 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:47.249057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:46469 (local address 127.12.78.62:44301)
0809 20:24:47.249262 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:47.249267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:47.249847 (+ 580us) server_negotiation.cc:407] Connection header received
0809 20:24:47.250004 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:47.250008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:47.250071 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:47.250166 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:47.251382 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:47.252396 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:47.253676 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:47.254117 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:47.254246 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:47.258633 (+ 4387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:46469: BlockingRecv error: recv got EOF from 127.12.78.3:46469 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 09 20:24:47 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:47 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:47.333137 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:47.324658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:48361)
0809 20:24:47.325209 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:47.325230 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:47.325396 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:47.325763 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:47.325768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:47.326201 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:47.326359 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:47.326369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:47.327421 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:47.327427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:47.328343 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:47.328354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:47.329193 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:47.329200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:47.329369 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:47.329968 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:47.329985 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:47.332904 (+ 2919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":324,"threads_started":1}
W20260809 20:24:47.333386 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:47.324776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:48361 (local address 127.12.78.62:44301)
0809 20:24:47.324987 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:47.324993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:47.325320 (+ 327us) server_negotiation.cc:407] Connection header received
0809 20:24:47.325523 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:47.325528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:47.325617 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:47.325726 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:47.326513 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:47.327269 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:47.328490 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:47.328996 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:47.329136 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:47.333144 (+ 4008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:48361: BlockingRecv error: recv got EOF from 127.12.78.1:48361 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 09 20:24:47 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:47 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:47.792703 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:47.784068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:51253)
0809 20:24:47.784713 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:47.784735 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:47.784879 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:47.785211 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:47.785216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:47.785642 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:47.785790 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:47.785798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:47.786945 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:47.786952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:47.787894 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:47.787905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:47.788786 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:47.788793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:47.788957 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:47.789517 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:47.789536 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:47.792322 (+ 2786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":275,"threads_started":1}
W20260809 20:24:47.792840 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:47.784147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:51253 (local address 127.12.78.62:44301)
0809 20:24:47.784399 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:47.784410 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:47.784803 (+ 393us) server_negotiation.cc:407] Connection header received
0809 20:24:47.784999 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:47.785004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:47.785081 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:47.785180 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:47.785938 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:47.786804 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:47.788054 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:47.788594 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:47.788746 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:47.792526 (+ 3780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:51253: BlockingRecv error: recv got EOF from 127.12.78.2:51253 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 20:24:48 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:48 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:48.268342 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:48.259750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:50563)
0809 20:24:48.260295 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:48.260311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:48.260474 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:48.260865 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:48.260870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:48.261323 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:48.261480 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:48.261488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:48.262501 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:48.262508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:48.263549 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:48.263565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:48.264321 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:48.264328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:48.264495 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:48.265219 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:48.265236 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:48.268109 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":268,"threads_started":1}
W20260809 20:24:48.268618 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:48.259903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:50563 (local address 127.12.78.62:44301)
0809 20:24:48.260126 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:48.260135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:48.260406 (+ 271us) server_negotiation.cc:407] Connection header received
0809 20:24:48.260654 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:48.260659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:48.260728 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:48.260833 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:48.261626 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:48.262350 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:48.263697 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:48.264138 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:48.264284 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:48.268304 (+ 4020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:50563: BlockingRecv error: recv got EOF from 127.12.78.3:50563 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 09 20:24:48 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:48 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:48.342994 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:48.334079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:40535)
0809 20:24:48.334714 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:48.334730 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:48.334894 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:48.335281 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:48.335286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:48.335847 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:48.336141 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:48.336159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:48.337289 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:48.337294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:48.338231 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:48.338242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:48.338972 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:48.338978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:48.339186 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:48.339748 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:48.339766 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:48.342724 (+ 2958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":338,"threads_started":1}
W20260809 20:24:48.343297 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:48.334211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:40535 (local address 127.12.78.62:44301)
0809 20:24:48.334473 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:48.334481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:48.334823 (+ 342us) server_negotiation.cc:407] Connection header received
0809 20:24:48.335032 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:48.335038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:48.335133 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:48.335238 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:48.336298 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:48.337149 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:48.338386 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:48.338819 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:48.338954 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:48.342939 (+ 3985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:40535: BlockingRecv error: recv got EOF from 127.12.78.1:40535 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 09 20:24:48 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:48 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:48.802058 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:48.793701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:44817)
0809 20:24:48.794195 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:48.794211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:48.794324 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:48.794644 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:48.794648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:48.795033 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:48.795217 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:48.795226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:48.796191 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:48.796195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:48.797387 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:48.797400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:48.798207 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:48.798212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:48.798339 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:48.798884 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:48.798900 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:48.801813 (+ 2913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":267,"threads_started":1}
W20260809 20:24:48.802248 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:48.793778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:44817 (local address 127.12.78.62:44301)
0809 20:24:48.794011 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:48.794018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:48.794311 (+ 293us) server_negotiation.cc:407] Connection header received
0809 20:24:48.794442 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:48.794446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:48.794510 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:48.794602 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:48.795340 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:48.796083 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:48.797523 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:48.798071 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:48.798166 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:48.802003 (+ 3837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:44817: BlockingRecv error: recv got EOF from 127.12.78.2:44817 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 09 20:24:49 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:49 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:49.277710 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:49.269224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:56021)
0809 20:24:49.269758 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:49.269775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:49.269940 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:49.270289 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:49.270294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:49.270757 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:49.270923 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:49.270932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:49.272034 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:49.272041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:49.273018 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:49.273029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:49.273780 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:49.273786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:49.273974 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:49.274542 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:49.274561 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:49.277445 (+ 2884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":277,"threads_started":1}
W20260809 20:24:49.277878 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:49.269331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:56021 (local address 127.12.78.62:44301)
0809 20:24:49.269598 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:49.269604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:49.269867 (+ 263us) server_negotiation.cc:407] Connection header received
0809 20:24:49.270069 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:49.270075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:49.270147 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:49.270267 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:49.271097 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:49.271879 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:49.273172 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:49.273608 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:49.273742 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:49.277648 (+ 3906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:56021: BlockingRecv error: recv got EOF from 127.12.78.3:56021 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 09 20:24:49 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:49 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:49.353078 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:49.344162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:41961)
0809 20:24:49.344819 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:49.344835 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:49.344952 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:49.345251 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:49.345254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:49.345646 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:49.345804 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:49.345813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:49.346797 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:49.346800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:49.347770 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:49.347781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:49.348581 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:49.348587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:49.348725 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:49.349468 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:49.349488 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:49.352749 (+ 3261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":465,"threads_started":1}
W20260809 20:24:49.353272 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:49.344238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:41961 (local address 127.12.78.62:44301)
0809 20:24:49.344419 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:49.344423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:49.344923 (+ 500us) server_negotiation.cc:407] Connection header received
0809 20:24:49.345054 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:49.345058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:49.345122 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:49.345202 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:49.345933 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:49.346680 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:49.347894 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:49.348342 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:49.348436 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:49.352945 (+ 4509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:41961: BlockingRecv error: recv got EOF from 127.12.78.1:41961 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 09 20:24:49 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:49 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:49.811748 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:49.803176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:49793)
0809 20:24:49.803742 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:49.803758 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:49.803873 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:49.804177 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:49.804181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:49.804630 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:49.804785 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:49.804794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:49.805886 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:49.805890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:49.806823 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:49.806835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:49.807602 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:49.807606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:49.807739 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:49.808347 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:49.808368 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:49.811427 (+ 3059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":327,"threads_started":1}
W20260809 20:24:49.811936 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:49.803217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:49793 (local address 127.12.78.62:44301)
0809 20:24:49.803417 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:49.803421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:49.803840 (+ 419us) server_negotiation.cc:407] Connection header received
0809 20:24:49.803975 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:49.803979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:49.804044 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:49.804131 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:49.804943 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:49.805757 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:49.807021 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:49.807457 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:49.807558 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:49.811682 (+ 4124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:49793: BlockingRecv error: recv got EOF from 127.12.78.2:49793 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 09 20:24:50 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:50 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:50.288754 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:50.278776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:53085)
0809 20:24:50.279359 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:50.279382 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:50.279567 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:50.279929 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:50.279933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:50.280501 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:50.280752 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:50.280765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:50.282180 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:50.282184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:50.283336 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:50.283350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:50.284293 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:50.284302 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:50.284481 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:50.285282 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:50.285300 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:50.288409 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":294,"threads_started":1}
W20260809 20:24:50.289284 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:50.279067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:53085 (local address 127.12.78.62:44301)
0809 20:24:50.279323 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:50.279329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:50.279487 (+ 158us) server_negotiation.cc:407] Connection header received
0809 20:24:50.279717 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:50.279721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:50.279786 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:50.279896 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:50.280934 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:50.282040 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:50.283538 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:50.284004 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:50.284138 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:50.288771 (+ 4633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:53085: BlockingRecv error: recv got EOF from 127.12.78.3:53085 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 09 20:24:50 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:50 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:50.362800 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:50.354330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:48729)
0809 20:24:50.355075 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:50.355097 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:50.355241 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:50.355606 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:50.355627 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:50.356040 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:50.356217 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:50.356225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:50.357247 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:50.357250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:50.358143 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:50.358155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:50.358901 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:50.358905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:50.359027 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:50.359698 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:50.359717 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:50.362552 (+ 2835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":471,"threads_started":1}
W20260809 20:24:50.363063 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:50.354375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:48729 (local address 127.12.78.62:44301)
0809 20:24:50.354660 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:50.354669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:50.355265 (+ 596us) server_negotiation.cc:407] Connection header received
0809 20:24:50.355359 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:50.355362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:50.355440 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:50.355540 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:50.356346 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:50.357132 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:50.358297 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:50.358768 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:50.358863 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:50.362777 (+ 3914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:48729: BlockingRecv error: recv got EOF from 127.12.78.1:48729 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 09 20:24:50 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:50 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:50.821388 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:50.812856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:59051)
0809 20:24:50.813459 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:50.813476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:50.813587 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:50.813865 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:50.813869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:50.814293 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:50.814467 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:50.814476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:50.815485 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:50.815489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:50.816368 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:50.816378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:50.817192 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:50.817196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:50.817319 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:50.817957 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:50.817975 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:50.821026 (+ 3051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":364,"threads_started":1}
W20260809 20:24:50.821513 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:50.812947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:59051 (local address 127.12.78.62:44301)
0809 20:24:50.813130 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:50.813134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:50.813596 (+ 462us) server_negotiation.cc:407] Connection header received
0809 20:24:50.813681 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:50.813684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:50.813746 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:50.813827 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:50.814622 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:50.815367 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:50.816504 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:50.817062 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:50.817156 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:50.821251 (+ 4095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:59051: BlockingRecv error: recv got EOF from 127.12.78.2:59051 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 20:24:51 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:51 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:51.303894 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:51.289785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:45777)
0809 20:24:51.290336 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:51.290356 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:51.290517 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:51.290920 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:51.290924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:51.291438 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:51.291629 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:51.291639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:51.292710 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:51.292716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:51.293754 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:51.293767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:51.294587 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:51.294596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:51.294758 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:51.299816 (+ 5058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:51.299838 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:51.303653 (+ 3815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":278,"threads_started":1}
W20260809 20:24:51.304090 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:51.289856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:45777 (local address 127.12.78.62:44301)
0809 20:24:51.290106 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:51.290114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:51.290476 (+ 362us) server_negotiation.cc:407] Connection header received
0809 20:24:51.290701 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:51.290706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:51.290777 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:51.290888 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:51.291782 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:51.292536 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:51.293896 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:51.294388 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:51.294515 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:51.303843 (+ 9328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:45777: BlockingRecv error: recv got EOF from 127.12.78.3:45777 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 20:24:51 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:51 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:51.372475 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:51.363937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:34603)
0809 20:24:51.364570 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:51.364586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:51.364702 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:51.365000 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:51.365004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:51.365438 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:51.365626 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:51.365636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:51.366645 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:51.366650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:51.367646 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:51.367661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:51.368425 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:51.368430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:51.368700 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:51.369347 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:51.369366 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:51.372238 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":358,"threads_started":1}
W20260809 20:24:51.372785 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:51.364058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:34603 (local address 127.12.78.62:44301)
0809 20:24:51.364255 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:51.364263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:51.364668 (+ 405us) server_negotiation.cc:407] Connection header received
0809 20:24:51.364797 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:51.364800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:51.364863 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:51.364945 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:51.365756 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:51.366471 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:51.367789 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:51.368260 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:51.368357 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:51.372454 (+ 4097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:34603: BlockingRecv error: recv got EOF from 127.12.78.1:34603 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 09 20:24:51 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:51 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:51.830950 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:51.822544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:45725)
0809 20:24:51.823164 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:51.823180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:51.823297 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:51.823630 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:51.823634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:51.824035 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:51.824212 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:51.824221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:51.825294 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:51.825298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:51.826212 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:51.826223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:51.826919 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:51.826923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:51.827058 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:51.827708 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:51.827724 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:51.830606 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":355,"threads_started":1}
W20260809 20:24:51.831053 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:51.822654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:45725 (local address 127.12.78.62:44301)
0809 20:24:51.822864 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:51.822868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:51.823260 (+ 392us) server_negotiation.cc:407] Connection header received
0809 20:24:51.823395 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:51.823398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:51.823462 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:51.823579 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:51.824339 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:51.825175 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:51.826350 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:51.826795 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:51.826898 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:51.830827 (+ 3929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:45725: BlockingRecv error: recv got EOF from 127.12.78.2:45725 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 20:24:52 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:52 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:52.313856 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:52.304968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:50033)
0809 20:24:52.305484 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:52.305561 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:52.305732 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:52.306121 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:52.306124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:52.306651 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:52.306808 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:52.306842 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:52.308094 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:52.308098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:52.309067 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:52.309079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:52.309785 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:52.309789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:52.309918 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:52.310612 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:52.310630 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:52.313624 (+ 2994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":310,"threads_started":1}
W20260809 20:24:52.314139 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:52.305016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:50033 (local address 127.12.78.62:44301)
0809 20:24:52.305299 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:52.305307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:52.305658 (+ 351us) server_negotiation.cc:407] Connection header received
0809 20:24:52.305871 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:52.305876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:52.305941 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:52.306060 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:52.306967 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:52.307935 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:52.309201 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:52.309648 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:52.309745 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:52.313841 (+ 4096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:50033: BlockingRecv error: recv got EOF from 127.12.78.3:50033 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 20:24:52 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:52 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:52.382289 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:52.373550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:58655)
0809 20:24:52.374133 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:52.374156 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:52.374276 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:52.374612 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:52.374616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:52.374993 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:52.375163 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:52.375173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:52.376133 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:52.376136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:52.377094 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:52.377105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:52.377876 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:52.377881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:52.378032 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:52.378585 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:52.378602 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:52.381923 (+ 3321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":299,"threads_started":1}
W20260809 20:24:52.382376 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:52.373606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:58655 (local address 127.12.78.62:44301)
0809 20:24:52.373827 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:52.373833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:52.374246 (+ 413us) server_negotiation.cc:407] Connection header received
0809 20:24:52.374376 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:52.374379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:52.374441 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:52.374527 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:52.375290 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:52.375998 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:52.377226 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:52.377686 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:52.377784 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:52.382129 (+ 4345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:58655: BlockingRecv error: recv got EOF from 127.12.78.1:58655 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 20:24:52 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:52 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:52.840883 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:52.832064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:34691)
0809 20:24:52.832676 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:52.832693 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:52.832817 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:52.833132 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:52.833136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:52.833579 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:52.833745 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:52.833755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:52.834750 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:52.834754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:52.835703 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:52.835715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:52.836408 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:52.836414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:52.836647 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:52.837253 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:52.837274 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:52.840634 (+ 3360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":286,"threads_started":1}
W20260809 20:24:52.841046 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:52.832269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:34691 (local address 127.12.78.62:44301)
0809 20:24:52.832493 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:52.832499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:52.832775 (+ 276us) server_negotiation.cc:407] Connection header received
0809 20:24:52.832917 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:52.832921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:52.832999 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:52.833084 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:52.833875 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:52.834633 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:52.835824 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:52.836218 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:52.836308 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:52.840804 (+ 4496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:34691: BlockingRecv error: recv got EOF from 127.12.78.2:34691 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 09 20:24:53 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:53 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:53.323827 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:53.314910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:38565)
0809 20:24:53.315455 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:53.315472 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:53.315661 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:53.315999 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:53.316005 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:53.316448 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:53.316678 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:53.316689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:53.317766 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:53.317772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:53.318761 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:53.318774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:53.319494 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:53.319500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:53.319679 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:53.320264 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:53.320283 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:53.323454 (+ 3171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":291,"threads_started":1}
W20260809 20:24:53.323962 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:53.314982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:38565 (local address 127.12.78.62:44301)
0809 20:24:53.315236 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:53.315244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:53.315584 (+ 340us) server_negotiation.cc:407] Connection header received
0809 20:24:53.315787 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:53.315799 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:53.315863 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:53.315968 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:53.316822 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:53.317622 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:53.318912 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:53.319332 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:53.319469 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:53.323716 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:38565: BlockingRecv error: recv got EOF from 127.12.78.3:38565 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 20:24:53 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:53 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:53.392973 22413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:53.383439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:40439)
0809 20:24:53.383954 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:53.383972 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:53.384098 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:53.384573 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:53.384595 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:53.384974 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:53.385183 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:53.385193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:53.386344 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:53.386347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:53.387358 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:53.387373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:53.388376 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:53.388383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:53.388620 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:53.389332 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:53.389352 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:53.392712 (+ 3360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":309,"threads_started":1}
W20260809 20:24:53.393272 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:53.383546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:40439 (local address 127.12.78.62:44301)
0809 20:24:53.383953 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:53.383963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:53.384076 (+ 113us) server_negotiation.cc:407] Connection header received
0809 20:24:53.384346 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:53.384351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:53.384418 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:53.384512 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:53.385331 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:53.386206 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:53.387509 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:53.388209 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:53.388359 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:53.392952 (+ 4593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:40439: BlockingRecv error: recv got EOF from 127.12.78.1:40439 (error 108)
Metrics: {"server-negotiator.queue_time_us":243}
Aug 09 20:24:53 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:53 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:53.850586 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:53.841937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:35453)
0809 20:24:53.842419 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:53.842436 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:53.842556 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:53.842881 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:53.842885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:53.843287 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:53.843470 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:53.843479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:53.844485 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:53.844489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:53.845505 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:53.845518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:53.846201 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:53.846205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:53.846325 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:53.846949 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:53.846967 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:53.850316 (+ 3349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":271,"threads_started":1}
W20260809 20:24:53.850749 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:53.842105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:35453 (local address 127.12.78.62:44301)
0809 20:24:53.842360 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:53.842366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:53.842520 (+ 154us) server_negotiation.cc:407] Connection header received
0809 20:24:53.842682 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:53.842686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:53.842748 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:53.842840 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:53.843629 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:53.844344 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:53.845640 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:53.846066 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:53.846166 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:53.850510 (+ 4344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:35453: BlockingRecv error: recv got EOF from 127.12.78.2:35453 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Aug 09 20:24:54 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:54 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.3@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:54.334025 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:54.324981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.3:55581)
0809 20:24:54.325562 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:54.325578 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:54.325745 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:54.326101 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:54.326106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:54.326510 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:54.326693 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:54.326702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:54.327808 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:54.327815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:54.328766 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:54.328778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:54.329530 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:54.329536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:54.329724 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:54.330402 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:54.330426 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:54.333794 (+ 3368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":273,"threads_started":1}
W20260809 20:24:54.334312 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:54.325067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:55581 (local address 127.12.78.62:44301)
0809 20:24:54.325338 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:54.325347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:54.325674 (+ 327us) server_negotiation.cc:407] Connection header received
0809 20:24:54.325877 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:54.325883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:54.325959 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:54.326074 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:54.326850 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:54.327652 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:54.328917 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:54.329367 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:54.329507 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:54.334031 (+ 4524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.3:55581: BlockingRecv error: recv got EOF from 127.12.78.3:55581 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 09 20:24:54 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:54 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.1@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:54.403142 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:54.394430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.1:59643)
0809 20:24:54.394990 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:54.395011 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:54.395152 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:54.395532 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:54.395536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:54.395971 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:54.396180 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:54.396189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:54.397256 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:54.397260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:54.398203 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:54.398216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:54.398962 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:54.398966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:54.399091 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:54.399690 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:54.399708 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:54.402870 (+ 3162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":299,"threads_started":1}
W20260809 20:24:54.403332 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:54.394485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:59643 (local address 127.12.78.62:44301)
0809 20:24:54.394757 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:54.394765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:54.395104 (+ 339us) server_negotiation.cc:407] Connection header received
0809 20:24:54.395252 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:54.395256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:54.395332 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:54.395433 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:54.396355 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:54.397117 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:54.398348 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:54.398824 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:54.398925 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:54.403104 (+ 4179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.1:59643: BlockingRecv error: recv got EOF from 127.12.78.1:59643 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 09 20:24:54 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 20:24:54 dist-test-slave-rnrw krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.78.2@KRBTEST.COM for oryx/127.12.78.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 20:24:54.861299 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:54.851844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44301 (local address 127.12.78.2:47517)
0809 20:24:54.852482 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:54.852503 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:54.852691 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:54.853069 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:54.853076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:54.853679 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:54.854040 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:54.854057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:54.855267 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:54.855274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:54.856219 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:54.856231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:54.857070 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:54.857077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:54.857254 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:54.857870 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:24:54.857887 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:54.860916 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.78.62:44301: Server oryx/127.12.78.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":343,"threads_started":1}
W20260809 20:24:54.861456 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:54.852047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:47517 (local address 127.12.78.62:44301)
0809 20:24:54.852282 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:54.852290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:54.852640 (+ 350us) server_negotiation.cc:407] Connection header received
0809 20:24:54.852828 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:54.852836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:54.852921 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:54.853008 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:54.854186 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:54.855103 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:54.856373 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:54.856916 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:54.857057 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:54.861179 (+ 4122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.78.2:47517: BlockingRecv error: recv got EOF from 127.12.78.2:47517 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 20:24:55.159669 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 21938
I20260809 20:24:55.173382 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:55.161001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.12.78.1:38971)
0809 20:24:55.161606 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:55.161615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:55.161642 (+ 27us) server_negotiation.cc:407] Connection header received
0809 20:24:55.161703 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:55.161709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:55.161939 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:55.162096 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:55.163783 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:55.164930 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:55.166252 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:55.166814 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:55.166921 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:55.170154 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:55.170163 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:55.170178 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:55.170218 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:55.172853 (+ 2635us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.78.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 20:24:55.173154 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48488: Request ticket server kudu/127.12.78.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":313,"threads_started":1}
W20260809 20:24:55.173521 22420 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48488: Request ticket server kudu/127.12.78.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 20:24:55.173791 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 21938: Not authorized: Client connection negotiation failed: client connection to 127.12.78.1:38971: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.78.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 20:24:55.174036 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 21938
I20260809 20:24:55.193753 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 22096
I20260809 20:24:55.206354 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:55.194957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.12.78.2:38087)
0809 20:24:55.195589 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:55.195597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:55.195626 (+ 29us) server_negotiation.cc:407] Connection header received
0809 20:24:55.195687 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:55.195694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:55.195910 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:55.196107 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:55.197419 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:55.198480 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:55.199652 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:55.200213 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:55.200313 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:55.202987 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:55.202995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:55.203009 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:55.203069 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:55.205763 (+ 2694us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.78.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 20:24:55.206067 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51440: Request ticket server kudu/127.12.78.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":308,"threads_started":1}
W20260809 20:24:55.206444 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 22096: Not authorized: Client connection negotiation failed: client connection to 127.12.78.2:38087: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.78.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 20:24:55.206476 22421 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51440: Request ticket server kudu/127.12.78.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 20:24:55.206621 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 22096
I20260809 20:24:55.225893 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 22233
I20260809 20:24:55.238539 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:55.226911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.12.78.3:40983)
0809 20:24:55.227498 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:55.227506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:55.227533 (+ 27us) server_negotiation.cc:407] Connection header received
0809 20:24:55.227601 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:55.227607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:55.227849 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:55.228005 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:55.229404 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:55.230485 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:55.231750 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:55.232302 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:55.232402 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:55.234963 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:55.234971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:55.234984 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:24:55.235027 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:55.237766 (+ 2739us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.78.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 20:24:55.238167 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59382: Request ticket server kudu/127.12.78.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":307,"threads_started":1}
W20260809 20:24:55.238746 22423 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59382: Request ticket server kudu/127.12.78.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 20:24:55.238772 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 22233: Not authorized: Client connection negotiation failed: client connection to 127.12.78.3:40983: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.78.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 20:24:55.238986 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 22233
I20260809 20:24:55.258782 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 21865
I20260809 20:24:55.260334 21931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:55.379542 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 21865
2026-08-09T20:24:55Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21223 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T20:24:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:24:55Z Disabled control of system clock
I20260809 20:24:55.470108 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:39307
--webserver_interface=127.12.78.62
--webserver_port=0
--builtin_ntp_servers=127.12.78.20:39359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 20:24:55.817669 22428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:55.817961 22428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:55.818041 22428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:55.825246 22428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:55.825363 22428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:55.825402 22428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:55.825460 22428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:55.825512 22428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:24:55.834789 22428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:39359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:39307
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:55.836115 22428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:55.837920 22428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:55.850088 22433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:55.850061 22436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:55.850183 22434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:55.850870 22428 server_base.cc:1034] running on GCE node
I20260809 20:24:55.851770 22428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:55.853308 22428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:55.854517 22428 hybrid_clock.cc:648] HybridClock initialized: now 1786307095854482 us; error 34 us; skew 500 ppm
I20260809 20:24:55.854892 22428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:55.857355 22428 webserver.cc:460] Webserver started at http://127.12.78.62:38461/ using document root <none> and password file <none>
I20260809 20:24:55.858093 22428 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:55.858213 22428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:55.858577 22428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:55.861756 22428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "57153aa3eb1941df862138c64a5448c0"
format_stamp: "Formatted at 2026-08-09 20:24:55 on dist-test-slave-rnrw"
server_key: "c0611e5f1edd72fc75283ada0129e942"
server_key_iv: "afe5e5e634e7d516c22fc64d17268629"
server_key_version: "encryptionkey@0"
I20260809 20:24:55.862636 22428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "57153aa3eb1941df862138c64a5448c0"
format_stamp: "Formatted at 2026-08-09 20:24:55 on dist-test-slave-rnrw"
server_key: "c0611e5f1edd72fc75283ada0129e942"
server_key_iv: "afe5e5e634e7d516c22fc64d17268629"
server_key_version: "encryptionkey@0"
I20260809 20:24:55.868319 22428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:24:55.871965 22442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:55.873479 22428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:55.873677 22428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "57153aa3eb1941df862138c64a5448c0"
format_stamp: "Formatted at 2026-08-09 20:24:55 on dist-test-slave-rnrw"
server_key: "c0611e5f1edd72fc75283ada0129e942"
server_key_iv: "afe5e5e634e7d516c22fc64d17268629"
server_key_version: "encryptionkey@0"
I20260809 20:24:55.873916 22428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:55.887391 22428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:55.888304 22428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:55.888690 22428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:55.907857 22428 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:39307
I20260809 20:24:55.907864 22496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:39307 every 8 connection(s)
I20260809 20:24:55.909447 22428 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:24:55.913478 22497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:55.917299 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 22428
I20260809 20:24:55.917519 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:24:55.918377 12600 external_mini_cluster.cc:1428] Setting key ea4b347534f758d65f0210f02b03c368
I20260809 20:24:55.927506 22497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0: Bootstrap starting.
I20260809 20:24:55.931285 22497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:55.932432 22497 log.cc:824] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:55.933187 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:55.920724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.12.78.62:39307)
0809 20:24:55.921724 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:55.921735 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:55.921776 (+ 41us) server_negotiation.cc:407] Connection header received
0809 20:24:55.923375 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:55.923397 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:55.923873 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:55.924224 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:55.925040 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:55.926391 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:55.928257 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:55.928998 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:55.929143 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:55.930460 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:55.930469 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:55.930485 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:55.931019 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:55.931109 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:55.931130 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:55.931220 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:55.931568 (+ 348us) server_negotiation.cc:299] Negotiation successful
0809 20:24:55.931917 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":377,"threads_started":1}
I20260809 20:24:55.935572 22497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0: No bootstrap required, opened a new log
I20260809 20:24:55.942747 22497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57153aa3eb1941df862138c64a5448c0" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39307 } }
I20260809 20:24:55.943135 22497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:55.943246 22497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57153aa3eb1941df862138c64a5448c0, State: Initialized, Role: FOLLOWER
I20260809 20:24:55.943848 22497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57153aa3eb1941df862138c64a5448c0" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39307 } }
I20260809 20:24:55.944101 22497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:24:55.944254 22497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:24:55.944424 22497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:55.946610 22497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57153aa3eb1941df862138c64a5448c0" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39307 } }
I20260809 20:24:55.947103 22497 leader_election.cc:302] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57153aa3eb1941df862138c64a5448c0; no voters:
I20260809 20:24:55.948082 22497 leader_election.cc:288] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:24:55.948256 22502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:55.949061 22502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [term 1 LEADER]: Becoming Leader. State: Replica: 57153aa3eb1941df862138c64a5448c0, State: Running, Role: LEADER
I20260809 20:24:55.949653 22502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57153aa3eb1941df862138c64a5448c0" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39307 } }
I20260809 20:24:55.950299 22497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:55.952734 22505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57153aa3eb1941df862138c64a5448c0. Latest consensus state: current_term: 1 leader_uuid: "57153aa3eb1941df862138c64a5448c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57153aa3eb1941df862138c64a5448c0" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39307 } } }
I20260809 20:24:55.953122 22505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:55.952822 22503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57153aa3eb1941df862138c64a5448c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57153aa3eb1941df862138c64a5448c0" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 39307 } } }
I20260809 20:24:55.953236 22503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0 [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:55.958339 22509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:55.963471 22509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:55.972213 22509 catalog_manager.cc:1347] Generated new cluster ID: 87621f21296148beaca1bd82bf8214fa
I20260809 20:24:55.972342 22509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:55.995575 22509 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:24:55.997298 22509 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:56.009399 22509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57153aa3eb1941df862138c64a5448c0: Generated new TSK 0
I20260809 20:24:56.010424 22509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:24:56.022569 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:39307
--builtin_ntp_servers=127.12.78.20:39359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 20:24:56.364264 22523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:56.364596 22523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:56.364774 22523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:56.372011 22523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:56.372104 22523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:56.372292 22523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:24:56.381783 22523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:39359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:39307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:56.383136 22523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:56.385217 22523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:56.398310 22528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:56.398778 22529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:56.399443 22532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:56.400029 22523 server_base.cc:1034] running on GCE node
I20260809 20:24:56.400688 22523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:56.401633 22523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:56.402842 22523 hybrid_clock.cc:648] HybridClock initialized: now 1786307096402800 us; error 31 us; skew 500 ppm
I20260809 20:24:56.403134 22523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:56.410197 22523 webserver.cc:460] Webserver started at http://127.12.78.1:37485/ using document root <none> and password file <none>
I20260809 20:24:56.410918 22523 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:56.411012 22523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:56.411305 22523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:56.414286 22523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "3e1c553920324024a2707c87292c467e"
format_stamp: "Formatted at 2026-08-09 20:24:56 on dist-test-slave-rnrw"
server_key: "a19a21d5a5c0e8797826fe38e69b7964"
server_key_iv: "724f5285bf48166843a2c8b5ce8d1f4e"
server_key_version: "encryptionkey@0"
I20260809 20:24:56.415057 22523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "3e1c553920324024a2707c87292c467e"
format_stamp: "Formatted at 2026-08-09 20:24:56 on dist-test-slave-rnrw"
server_key: "a19a21d5a5c0e8797826fe38e69b7964"
server_key_iv: "724f5285bf48166843a2c8b5ce8d1f4e"
server_key_version: "encryptionkey@0"
I20260809 20:24:56.420384 22523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 20:24:56.424371 22539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:56.426215 22523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:56.426434 22523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "3e1c553920324024a2707c87292c467e"
format_stamp: "Formatted at 2026-08-09 20:24:56 on dist-test-slave-rnrw"
server_key: "a19a21d5a5c0e8797826fe38e69b7964"
server_key_iv: "724f5285bf48166843a2c8b5ce8d1f4e"
server_key_version: "encryptionkey@0"
I20260809 20:24:56.426703 22523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:56.443791 22523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:56.444801 22523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:56.445112 22523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:56.446154 22523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:56.448007 22523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:56.448150 22523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:56.448310 22523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:56.448421 22523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:56.490927 22523 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:32973
I20260809 20:24:56.490971 22657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:32973 every 8 connection(s)
I20260809 20:24:56.492439 22523 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:24:56.501832 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 22523
I20260809 20:24:56.502156 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:24:56.502842 12600 external_mini_cluster.cc:1428] Setting key 8bb00bff8feac253520cd412ccb1534e
I20260809 20:24:56.505628 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:39307
--builtin_ntp_servers=127.12.78.20:39359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 20:24:56.506704 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:56.495239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:53211 (local address 127.12.78.62:39307)
0809 20:24:56.495770 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:56.495776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:56.497116 (+ 1340us) server_negotiation.cc:407] Connection header received
0809 20:24:56.499214 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:56.499220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:56.499324 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:56.499460 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:56.500681 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:56.501517 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:56.503451 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:56.503945 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:56.504098 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:56.505728 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:56.505740 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:56.505743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:56.505785 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:56.505839 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:56.505856 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:56.505963 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:56.506213 (+ 250us) server_negotiation.cc:299] Negotiation successful
0809 20:24:56.506323 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":307,"threads_started":1}
I20260809 20:24:56.507869 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:56.495735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39307 (local address 127.12.78.1:53211)
0809 20:24:56.496929 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:56.496984 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:56.498897 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:56.499731 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:56.499739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:56.499833 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:56.500469 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:56.500487 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:56.501694 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:56.501700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:56.503286 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:56.503300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:56.504207 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:56.504219 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:56.504491 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:56.505362 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:56.505365 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:56.505408 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:56.505511 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:56.505954 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:56.505967 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:56.505978 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 20:24:56.506185 (+ 207us) client_negotiation.cc:240] Negotiation successful
0809 20:24:56.506652 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":400,"threads_started":1}
I20260809 20:24:56.509626 22658 heartbeater.cc:347] Connected to a master server at 127.12.78.62:39307
I20260809 20:24:56.509979 22658 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:56.511281 22658 heartbeater.cc:511] Master 127.12.78.62:39307 requested a full tablet report, sending...
I20260809 20:24:56.513988 22461 ts_manager.cc:194] Registered new tserver with Master: 3e1c553920324024a2707c87292c467e (127.12.78.1:32973)
I20260809 20:24:56.515866 22461 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.78.1:53211
W20260809 20:24:56.863663 22662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:56.864140 22662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:56.864302 22662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:56.871913 22662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:56.872035 22662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:56.872213 22662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:24:56.882118 22662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:39359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:39307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:56.883576 22662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:56.885550 22662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:56.900377 22667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:56.900408 22670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:56.900431 22668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:56.901398 22662 server_base.cc:1034] running on GCE node
I20260809 20:24:56.902079 22662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:56.903123 22662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:56.904341 22662 hybrid_clock.cc:648] HybridClock initialized: now 1786307096904259 us; error 70 us; skew 500 ppm
I20260809 20:24:56.904874 22662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:56.907457 22662 webserver.cc:460] Webserver started at http://127.12.78.2:36865/ using document root <none> and password file <none>
I20260809 20:24:56.908234 22662 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:56.908381 22662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:56.908812 22662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:56.911808 22662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "117080f590f841f8ba849cdead282d25"
format_stamp: "Formatted at 2026-08-09 20:24:56 on dist-test-slave-rnrw"
server_key: "f2f3dfe39f916970e11d73bd33eb0100"
server_key_iv: "080b01f888b357067f3ca8849ef638b8"
server_key_version: "encryptionkey@0"
I20260809 20:24:56.912758 22662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "117080f590f841f8ba849cdead282d25"
format_stamp: "Formatted at 2026-08-09 20:24:56 on dist-test-slave-rnrw"
server_key: "f2f3dfe39f916970e11d73bd33eb0100"
server_key_iv: "080b01f888b357067f3ca8849ef638b8"
server_key_version: "encryptionkey@0"
I20260809 20:24:56.918326 22662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:24:56.922076 22677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:56.923614 22662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:24:56.923839 22662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "117080f590f841f8ba849cdead282d25"
format_stamp: "Formatted at 2026-08-09 20:24:56 on dist-test-slave-rnrw"
server_key: "f2f3dfe39f916970e11d73bd33eb0100"
server_key_iv: "080b01f888b357067f3ca8849ef638b8"
server_key_version: "encryptionkey@0"
I20260809 20:24:56.924080 22662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:56.959790 22662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:56.960870 22662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:56.961198 22662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:56.962324 22662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:56.963881 22662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:56.964010 22662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:56.964164 22662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:56.964257 22662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:56.999950 22662 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:45865
I20260809 20:24:57.000061 22797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:45865 every 8 connection(s)
I20260809 20:24:57.001495 22662 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:24:57.005939 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 22662
I20260809 20:24:57.006131 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:24:57.006605 12600 external_mini_cluster.cc:1428] Setting key d8d9f5c9b5bb435acb37599719c12b2a
I20260809 20:24:57.009943 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:39307
--builtin_ntp_servers=127.12.78.20:39359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 20:24:57.015129 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:57.004278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:57979 (local address 127.12.78.62:39307)
0809 20:24:57.004490 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:57.004495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:57.006022 (+ 1527us) server_negotiation.cc:407] Connection header received
0809 20:24:57.008256 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:57.008261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:57.008330 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:57.008455 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:57.009652 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.010489 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.012131 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.012623 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.012764 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:57.014272 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:57.014277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:57.014281 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:57.014302 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:57.014334 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:57.014347 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:57.014519 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:57.014702 (+ 183us) server_negotiation.cc:299] Negotiation successful
0809 20:24:57.014811 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 20:24:57.016243 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:57.004868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39307 (local address 127.12.78.2:57979)
0809 20:24:57.005829 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:57.005866 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:57.007956 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:57.008773 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:57.008784 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:57.008902 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:57.009477 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:57.009493 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.010657 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.010661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:57.011973 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:57.011985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.012876 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.012884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:57.013125 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:57.013892 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:57.013895 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:57.013938 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:57.014043 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:57.014450 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:57.014462 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:57.014470 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 20:24:57.014692 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 20:24:57.015141 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":312,"threads_started":1}
I20260809 20:24:57.017783 22798 heartbeater.cc:347] Connected to a master server at 127.12.78.62:39307
I20260809 20:24:57.018113 22798 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:57.018980 22798 heartbeater.cc:511] Master 127.12.78.62:39307 requested a full tablet report, sending...
I20260809 20:24:57.020488 22461 ts_manager.cc:194] Registered new tserver with Master: 117080f590f841f8ba849cdead282d25 (127.12.78.2:45865)
I20260809 20:24:57.021378 22461 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.78.2:57979
W20260809 20:24:57.353423 22801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:57.353842 22801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:57.354013 22801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:57.361622 22801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:57.361719 22801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:57.361822 22801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:24:57.370922 22801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:39359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:39307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:57.372648 22801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:57.374361 22801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:57.387704 22806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:57.387712 22807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:57.387733 22809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:57.388824 22801 server_base.cc:1034] running on GCE node
I20260809 20:24:57.389536 22801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:57.390547 22801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:57.391794 22801 hybrid_clock.cc:648] HybridClock initialized: now 1786307097391746 us; error 41 us; skew 500 ppm
I20260809 20:24:57.392128 22801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:57.394698 22801 webserver.cc:460] Webserver started at http://127.12.78.3:43055/ using document root <none> and password file <none>
I20260809 20:24:57.395484 22801 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:57.395579 22801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:57.395854 22801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:57.398838 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "dcfaa2548ad746c5911dcea5eb43a06c"
format_stamp: "Formatted at 2026-08-09 20:24:57 on dist-test-slave-rnrw"
server_key: "32d06d56e0aa6781aae240a2fd5ba32a"
server_key_iv: "8c2be20a29d6f56464950f826fbba05e"
server_key_version: "encryptionkey@0"
I20260809 20:24:57.399643 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "dcfaa2548ad746c5911dcea5eb43a06c"
format_stamp: "Formatted at 2026-08-09 20:24:57 on dist-test-slave-rnrw"
server_key: "32d06d56e0aa6781aae240a2fd5ba32a"
server_key_iv: "8c2be20a29d6f56464950f826fbba05e"
server_key_version: "encryptionkey@0"
I20260809 20:24:57.404937 22801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 20:24:57.408355 22816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:57.409791 22801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:57.409983 22801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "dcfaa2548ad746c5911dcea5eb43a06c"
format_stamp: "Formatted at 2026-08-09 20:24:57 on dist-test-slave-rnrw"
server_key: "32d06d56e0aa6781aae240a2fd5ba32a"
server_key_iv: "8c2be20a29d6f56464950f826fbba05e"
server_key_version: "encryptionkey@0"
I20260809 20:24:57.410218 22801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:57.433923 22801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:57.435276 22801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:57.435601 22801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:57.436721 22801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:57.438293 22801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:57.438409 22801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:57.438550 22801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:57.438655 22801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:57.488695 22801 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:42075
I20260809 20:24:57.488760 22940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:42075 every 8 connection(s)
I20260809 20:24:57.490629 22801 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:24:57.499385 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 22801
I20260809 20:24:57.499579 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:24:57.500229 12600 external_mini_cluster.cc:1428] Setting key 18fa477cca804dab80c86a88d7718900
I20260809 20:24:57.506207 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:57.493613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:50171 (local address 127.12.78.62:39307)
0809 20:24:57.493791 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:57.493796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:57.495813 (+ 2017us) server_negotiation.cc:407] Connection header received
0809 20:24:57.497920 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:57.497927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:57.498040 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:57.498166 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:57.499415 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.500520 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.502779 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.503253 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.503401 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:57.505304 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:57.505318 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:57.505321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:57.505341 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:57.505373 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:57.505385 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:57.505490 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:57.505732 (+ 242us) server_negotiation.cc:299] Negotiation successful
0809 20:24:57.505836 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 20:24:57.507282 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:57.494459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:39307 (local address 127.12.78.3:50171)
0809 20:24:57.495629 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:57.495668 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:57.497619 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:57.498426 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:57.498446 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:57.498598 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:57.499228 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:57.499245 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.500725 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.500730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:57.502620 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:24:57.502633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.503505 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.503513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:24:57.503870 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:57.504946 (+ 1076us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:57.504950 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:57.505000 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:57.505087 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:57.505496 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:57.505509 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:57.505519 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 20:24:57.505719 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 20:24:57.506147 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":441,"threads_started":1}
I20260809 20:24:57.508808 22941 heartbeater.cc:347] Connected to a master server at 127.12.78.62:39307
I20260809 20:24:57.509155 22941 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:57.509923 22941 heartbeater.cc:511] Master 127.12.78.62:39307 requested a full tablet report, sending...
I20260809 20:24:57.511415 22461 ts_manager.cc:194] Registered new tserver with Master: dcfaa2548ad746c5911dcea5eb43a06c (127.12.78.3:42075)
I20260809 20:24:57.512223 22461 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.78.3:50171
I20260809 20:24:57.517321 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:24:57.519048 22658 heartbeater.cc:503] Master 127.12.78.62:39307 was elected leader, sending a full tablet report...
W20260809 20:24:57.524389 22660 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:57.522559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.12.78.62:39307)
0809 20:24:57.522784 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:57.522792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:57.523359 (+ 567us) server_negotiation.cc:407] Connection header received
0809 20:24:57.523530 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:57.523534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:57.523595 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:57.523683 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:57.524126 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37894: BlockingRecv error: recv got EOF from 127.0.0.1:37894 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 20:24:57.525418 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 22523
I20260809 20:24:57.535776 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:57.526633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.12.78.1:32973)
0809 20:24:57.527215 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:57.527222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:57.527247 (+ 25us) server_negotiation.cc:407] Connection header received
0809 20:24:57.527322 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:57.527328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:57.527551 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:57.527676 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:57.528201 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.529518 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.531762 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.532436 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.532673 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:57.533982 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:57.533991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:57.534004 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:57.534630 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:57.534680 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:57.534700 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:57.534789 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:57.534992 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 20:24:57.535395 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":360,"threads_started":1}
I20260809 20:24:57.537150 22653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:57.621706 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 22523
I20260809 20:24:57.640249 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 22662
I20260809 20:24:57.647989 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:57.641209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.12.78.2:45865)
0809 20:24:57.641746 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:57.641753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:57.641780 (+ 27us) server_negotiation.cc:407] Connection header received
0809 20:24:57.641838 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:57.641845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:57.642052 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:57.642183 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:57.642576 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.643557 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.644990 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.645552 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.645649 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:57.646630 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:57.646638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:57.646651 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:57.647226 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:57.647284 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:57.647303 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:57.647392 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:57.647553 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 20:24:57.647667 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":265,"threads_started":1}
I20260809 20:24:57.649220 22793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:57.730298 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 22662
I20260809 20:24:57.748015 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 22801
I20260809 20:24:57.757072 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:57.749279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.12.78.3:42075)
0809 20:24:57.749909 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:57.749916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:57.749948 (+ 32us) server_negotiation.cc:407] Connection header received
0809 20:24:57.750004 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:57.750010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:57.750259 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:57.750388 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:57.750984 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.752106 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.753386 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:24:57.753936 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:24:57.754039 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:24:57.755479 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:57.755487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:57.755500 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:57.756131 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:57.756175 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:57.756195 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:57.756282 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:57.756487 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 20:24:57.756659 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":330,"threads_started":1}
I20260809 20:24:57.758491 22936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:57.846889 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 22801
I20260809 20:24:57.865129 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 22428
I20260809 20:24:57.866521 22492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:24:57.973196 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 22428
2026-08-09T20:24:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2588 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T20:24:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:24:58Z Disabled control of system clock
I20260809 20:24:58.056341 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44071
--webserver_interface=127.12.78.62
--webserver_port=0
--builtin_ntp_servers=127.12.78.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:44071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 20:24:58.394575 22960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:58.394892 22960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:58.395058 22960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:58.402799 22960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:24:58.402915 22960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:58.403014 22960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:58.403133 22960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:24:58.403218 22960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:24:58.412717 22960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:44071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:58.414052 22960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:58.415907 22960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:58.427991 22966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:58.428107 22965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:58.428076 22968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:58.429219 22960 server_base.cc:1034] running on GCE node
I20260809 20:24:58.430033 22960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:58.431450 22960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:58.432655 22960 hybrid_clock.cc:648] HybridClock initialized: now 1786307098432596 us; error 49 us; skew 500 ppm
I20260809 20:24:58.432978 22960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:58.435217 22960 webserver.cc:460] Webserver started at http://127.12.78.62:42693/ using document root <none> and password file <none>
I20260809 20:24:58.436157 22960 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:58.436295 22960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:58.436712 22960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:58.440407 22960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "da8ceb3de0b94db9a8d823bfbfc7e65c"
format_stamp: "Formatted at 2026-08-09 20:24:58 on dist-test-slave-rnrw"
server_key: "933695517e3fe4a23094bf2b8a8a41be"
server_key_iv: "3f29408f0455ee41070938358039ec61"
server_key_version: "encryptionkey@0"
I20260809 20:24:58.441316 22960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "da8ceb3de0b94db9a8d823bfbfc7e65c"
format_stamp: "Formatted at 2026-08-09 20:24:58 on dist-test-slave-rnrw"
server_key: "933695517e3fe4a23094bf2b8a8a41be"
server_key_iv: "3f29408f0455ee41070938358039ec61"
server_key_version: "encryptionkey@0"
I20260809 20:24:58.446587 22960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260809 20:24:58.450215 22976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:58.451570 22960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 20:24:58.451763 22960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "da8ceb3de0b94db9a8d823bfbfc7e65c"
format_stamp: "Formatted at 2026-08-09 20:24:58 on dist-test-slave-rnrw"
server_key: "933695517e3fe4a23094bf2b8a8a41be"
server_key_iv: "3f29408f0455ee41070938358039ec61"
server_key_version: "encryptionkey@0"
I20260809 20:24:58.452001 22960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:58.458070 22960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:58.458873 22960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:58.459187 22960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:58.478277 22960 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:44071
I20260809 20:24:58.478312 23032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:44071 every 8 connection(s)
I20260809 20:24:58.479736 22960 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:24:58.480383 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 22960
I20260809 20:24:58.480706 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:24:58.481488 12600 external_mini_cluster.cc:1428] Setting key b91cbf7b5415ce881abe9501a0a06b94
I20260809 20:24:58.484355 23033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:24:58.490149 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:58.483577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56278 (local address 127.12.78.62:44071)
0809 20:24:58.484738 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:58.484750 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:58.484794 (+ 44us) server_negotiation.cc:407] Connection header received
0809 20:24:58.486390 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:58.486411 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:58.486845 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:58.487279 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:58.488423 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:58.488428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:58.488443 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:58.488462 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:58.488531 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:58.488592 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:58.488839 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:58.488985 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 20:24:58.489245 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":403,"threads_started":1}
I20260809 20:24:58.496382 23033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c: Bootstrap starting.
I20260809 20:24:58.500128 23033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c: Neither blocks nor log segments found. Creating new log.
I20260809 20:24:58.501366 23033 log.cc:824] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c: Log is configured to *not* fsync() on all Append() calls
I20260809 20:24:58.504401 23033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c: No bootstrap required, opened a new log
I20260809 20:24:58.511878 23033 raft_consensus.cc:348] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8ceb3de0b94db9a8d823bfbfc7e65c" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44071 } }
I20260809 20:24:58.512238 23033 raft_consensus.cc:374] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:24:58.512312 23033 raft_consensus.cc:729] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8ceb3de0b94db9a8d823bfbfc7e65c, State: Initialized, Role: FOLLOWER
I20260809 20:24:58.512907 23033 consensus_queue.cc:260] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8ceb3de0b94db9a8d823bfbfc7e65c" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44071 } }
I20260809 20:24:58.513171 23033 raft_consensus.cc:388] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:24:58.513335 23033 raft_consensus.cc:482] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:24:58.513541 23033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:24:58.515697 23033 raft_consensus.cc:504] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8ceb3de0b94db9a8d823bfbfc7e65c" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44071 } }
I20260809 20:24:58.516209 23033 leader_election.cc:302] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da8ceb3de0b94db9a8d823bfbfc7e65c; no voters:
I20260809 20:24:58.516930 23033 leader_election.cc:288] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:24:58.517087 23039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:24:58.517861 23039 raft_consensus.cc:686] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [term 1 LEADER]: Becoming Leader. State: Replica: da8ceb3de0b94db9a8d823bfbfc7e65c, State: Running, Role: LEADER
I20260809 20:24:58.518501 23039 consensus_queue.cc:237] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8ceb3de0b94db9a8d823bfbfc7e65c" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44071 } }
I20260809 20:24:58.519140 23033 sys_catalog.cc:564] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:24:58.521363 23041 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [sys.catalog]: SysCatalogTable state changed. Reason: New leader da8ceb3de0b94db9a8d823bfbfc7e65c. Latest consensus state: current_term: 1 leader_uuid: "da8ceb3de0b94db9a8d823bfbfc7e65c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8ceb3de0b94db9a8d823bfbfc7e65c" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44071 } } }
I20260809 20:24:58.521757 23041 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:58.523296 23040 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da8ceb3de0b94db9a8d823bfbfc7e65c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8ceb3de0b94db9a8d823bfbfc7e65c" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44071 } } }
I20260809 20:24:58.523741 23040 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c [sys.catalog]: This master's current role is: LEADER
I20260809 20:24:58.524824 23046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:24:58.530045 23046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:24:58.538681 23046 catalog_manager.cc:1347] Generated new cluster ID: 08a13b26e4a147a1b9e3e99a4796d9b8
I20260809 20:24:58.538793 23046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:24:58.566232 23046 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:24:58.566773 23046 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:24:58.577922 23046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da8ceb3de0b94db9a8d823bfbfc7e65c: Generated new TSK 0
I20260809 20:24:58.578809 23046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:24:58.591725 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44071
--builtin_ntp_servers=127.12.78.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 20:24:58.941403 23060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:58.941673 23060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:58.941838 23060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:58.949093 23060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:58.949186 23060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:58.949358 23060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:24:58.958249 23060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:58.959524 23060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:58.961375 23060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:58.975068 23068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:58.975103 23066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:58.975032 23065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:58.975991 23060 server_base.cc:1034] running on GCE node
I20260809 20:24:58.976714 23060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:58.977730 23060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:58.979000 23060 hybrid_clock.cc:648] HybridClock initialized: now 1786307098978955 us; error 66 us; skew 500 ppm
I20260809 20:24:58.979328 23060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:58.981714 23060 webserver.cc:460] Webserver started at http://127.12.78.1:34581/ using document root <none> and password file <none>
I20260809 20:24:58.982437 23060 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:58.982581 23060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:58.982910 23060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:58.985841 23060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "fd97fc67aff5451da457542f71fc6146"
format_stamp: "Formatted at 2026-08-09 20:24:58 on dist-test-slave-rnrw"
server_key: "98c552aa3166d73b96203260ec5fcc59"
server_key_iv: "16a6d4c920f1c1e36d24ee9d95a20cb8"
server_key_version: "encryptionkey@0"
I20260809 20:24:58.986660 23060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "fd97fc67aff5451da457542f71fc6146"
format_stamp: "Formatted at 2026-08-09 20:24:58 on dist-test-slave-rnrw"
server_key: "98c552aa3166d73b96203260ec5fcc59"
server_key_iv: "16a6d4c920f1c1e36d24ee9d95a20cb8"
server_key_version: "encryptionkey@0"
I20260809 20:24:58.992076 23060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:24:58.995669 23074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:58.997123 23060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:58.997303 23060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "fd97fc67aff5451da457542f71fc6146"
format_stamp: "Formatted at 2026-08-09 20:24:58 on dist-test-slave-rnrw"
server_key: "98c552aa3166d73b96203260ec5fcc59"
server_key_iv: "16a6d4c920f1c1e36d24ee9d95a20cb8"
server_key_version: "encryptionkey@0"
I20260809 20:24:58.997577 23060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:59.003401 23060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:59.004242 23060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:59.004523 23060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:59.005496 23060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:59.006994 23060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:59.007125 23060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:59.007253 23060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:59.007342 23060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:59.042877 23060 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:38563
I20260809 20:24:59.042932 23191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:38563 every 8 connection(s)
I20260809 20:24:59.044201 23060 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:24:59.050807 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 23060
I20260809 20:24:59.051033 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:24:59.051644 12600 external_mini_cluster.cc:1428] Setting key b2ef78801b4cfd11bc0a184ac675e673
I20260809 20:24:59.053397 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:59.046828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:55659 (local address 127.12.78.62:44071)
0809 20:24:59.047361 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:59.047365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:59.048633 (+ 1268us) server_negotiation.cc:407] Connection header received
0809 20:24:59.050484 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:59.050490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:59.050575 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:59.050706 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:59.052217 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:59.052223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:59.052228 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:59.052258 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:59.052303 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:59.052316 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:59.052477 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:59.052786 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 20:24:59.052889 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":308,"threads_started":1}
I20260809 20:24:59.054232 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:59.047188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44071 (local address 127.12.78.1:55659)
0809 20:24:59.048325 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:59.048369 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:59.050190 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:59.050988 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:59.051006 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:59.051121 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:59.051908 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:59.051911 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:59.051948 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:59.052034 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:59.052479 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:59.052490 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:59.052498 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 20:24:59.052837 (+ 339us) client_negotiation.cc:240] Negotiation successful
0809 20:24:59.053110 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":463,"threads_started":1}
I20260809 20:24:59.055047 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44071
--builtin_ntp_servers=127.12.78.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 20:24:59.055697 23192 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44071
I20260809 20:24:59.055989 23192 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:59.056607 23192 heartbeater.cc:511] Master 127.12.78.62:44071 requested a full tablet report, sending...
I20260809 20:24:59.058578 22994 ts_manager.cc:194] Registered new tserver with Master: fd97fc67aff5451da457542f71fc6146 (127.12.78.1:38563)
W20260809 20:24:59.395342 23196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:59.395637 23196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:59.395728 23196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:59.403321 23196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:59.403416 23196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:59.403527 23196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:24:59.412699 23196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:59.413923 23196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:59.415660 23196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:59.429208 23204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:59.429303 23202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:59.429247 23201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:59.434149 23196 server_base.cc:1034] running on GCE node
I20260809 20:24:59.434909 23196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:59.436066 23196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:59.437316 23196 hybrid_clock.cc:648] HybridClock initialized: now 1786307099437278 us; error 50 us; skew 500 ppm
I20260809 20:24:59.437665 23196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:59.440616 23196 webserver.cc:460] Webserver started at http://127.12.78.2:43873/ using document root <none> and password file <none>
I20260809 20:24:59.441672 23196 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:59.441803 23196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:59.442227 23196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:59.445420 23196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "c18802a061974546931a3d0655e7cc82"
format_stamp: "Formatted at 2026-08-09 20:24:59 on dist-test-slave-rnrw"
server_key: "6adcb8ac5b077011e97c686ebfa68d55"
server_key_iv: "e2fc74331cb32227dac4bbc638c2e81c"
server_key_version: "encryptionkey@0"
I20260809 20:24:59.446171 23196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "c18802a061974546931a3d0655e7cc82"
format_stamp: "Formatted at 2026-08-09 20:24:59 on dist-test-slave-rnrw"
server_key: "6adcb8ac5b077011e97c686ebfa68d55"
server_key_iv: "e2fc74331cb32227dac4bbc638c2e81c"
server_key_version: "encryptionkey@0"
I20260809 20:24:59.451566 23196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:24:59.455242 23210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:59.456741 23196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:24:59.456895 23196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "c18802a061974546931a3d0655e7cc82"
format_stamp: "Formatted at 2026-08-09 20:24:59 on dist-test-slave-rnrw"
server_key: "6adcb8ac5b077011e97c686ebfa68d55"
server_key_iv: "e2fc74331cb32227dac4bbc638c2e81c"
server_key_version: "encryptionkey@0"
I20260809 20:24:59.457147 23196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:59.463011 23196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:59.463801 23196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:59.464075 23196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:59.465296 23196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:59.466827 23196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:59.466938 23196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:59.467072 23196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:59.467163 23196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:59.502661 23196 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:39239
I20260809 20:24:59.502748 23333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:39239 every 8 connection(s)
I20260809 20:24:59.504055 23196 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:24:59.512816 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:59.506609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:45763 (local address 127.12.78.62:44071)
0809 20:24:59.506848 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:59.506855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:59.508291 (+ 1436us) server_negotiation.cc:407] Connection header received
0809 20:24:59.509970 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:59.509974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:59.510048 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:59.510140 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:59.511369 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:59.511373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:59.511376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:59.511393 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:59.511424 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:59.511435 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:59.511536 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:59.512086 (+ 550us) server_negotiation.cc:299] Negotiation successful
0809 20:24:59.512177 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 20:24:59.512888 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 23196
I20260809 20:24:59.513422 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:24:59.514070 12600 external_mini_cluster.cc:1428] Setting key 40f69286712d5a3bc3564244958ca77f
I20260809 20:24:59.514057 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:59.507208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44071 (local address 127.12.78.2:45763)
0809 20:24:59.508139 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:59.508175 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:59.509725 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:59.510347 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:59.510355 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:59.510440 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:59.511076 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:59.511079 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:59.511129 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:59.511213 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:59.511568 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:59.511591 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:59.511599 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 20:24:59.512031 (+ 432us) client_negotiation.cc:240] Negotiation successful
0809 20:24:59.512376 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":304,"threads_started":1}
I20260809 20:24:59.515538 23334 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44071
I20260809 20:24:59.515904 23334 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:59.516494 23334 heartbeater.cc:511] Master 127.12.78.62:44071 requested a full tablet report, sending...
I20260809 20:24:59.516945 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44071
--builtin_ntp_servers=127.12.78.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 20:24:59.517984 22994 ts_manager.cc:194] Registered new tserver with Master: c18802a061974546931a3d0655e7cc82 (127.12.78.2:39239)
W20260809 20:24:59.864578 23338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:24:59.864885 23338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:24:59.864979 23338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:24:59.872243 23338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:24:59.872365 23338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:24:59.872480 23338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:24:59.881953 23338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:24:59.883242 23338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:24:59.885072 23338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:24:59.899565 23344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:59.899451 23347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:24:59.899724 23345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:24:59.900959 23338 server_base.cc:1034] running on GCE node
I20260809 20:24:59.901691 23338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:24:59.902743 23338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:24:59.903946 23338 hybrid_clock.cc:648] HybridClock initialized: now 1786307099903873 us; error 60 us; skew 500 ppm
I20260809 20:24:59.904315 23338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:24:59.906672 23338 webserver.cc:460] Webserver started at http://127.12.78.3:43871/ using document root <none> and password file <none>
I20260809 20:24:59.907395 23338 fs_manager.cc:359] Metadata directory not provided
I20260809 20:24:59.907531 23338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:24:59.907871 23338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:24:59.910913 23338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "294b5970e4284d2f87402c724a7e5a6b"
format_stamp: "Formatted at 2026-08-09 20:24:59 on dist-test-slave-rnrw"
server_key: "82c5d479a32bf905084a504d875fc6de"
server_key_iv: "2e762044aea6db23c64a093687b6bc41"
server_key_version: "encryptionkey@0"
I20260809 20:24:59.911767 23338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "294b5970e4284d2f87402c724a7e5a6b"
format_stamp: "Formatted at 2026-08-09 20:24:59 on dist-test-slave-rnrw"
server_key: "82c5d479a32bf905084a504d875fc6de"
server_key_iv: "2e762044aea6db23c64a093687b6bc41"
server_key_version: "encryptionkey@0"
I20260809 20:24:59.917253 23338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260809 20:24:59.920837 23353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:59.922230 23338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:24:59.922406 23338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "294b5970e4284d2f87402c724a7e5a6b"
format_stamp: "Formatted at 2026-08-09 20:24:59 on dist-test-slave-rnrw"
server_key: "82c5d479a32bf905084a504d875fc6de"
server_key_iv: "2e762044aea6db23c64a093687b6bc41"
server_key_version: "encryptionkey@0"
I20260809 20:24:59.922636 23338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:24:59.928819 23338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:24:59.929656 23338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:24:59.929929 23338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:24:59.930887 23338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:24:59.932309 23338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:24:59.932420 23338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:24:59.932569 23338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:24:59.932689 23338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260809 20:24:59.966100 23338 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:39255
I20260809 20:24:59.966190 23477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:39255 every 8 connection(s)
I20260809 20:24:59.967495 23338 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:24:59.974416 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 23338
I20260809 20:24:59.974635 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:24:59.975096 12600 external_mini_cluster.cc:1428] Setting key a8effe538901d32f22607a67ad75ecf4
I20260809 20:24:59.976087 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:59.970186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:39853 (local address 127.12.78.62:44071)
0809 20:24:59.970369 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:59.970373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:59.971915 (+ 1542us) server_negotiation.cc:407] Connection header received
0809 20:24:59.973714 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:59.973720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:59.973791 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:59.973907 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:59.975264 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:24:59.975270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:24:59.975274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:24:59.975295 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:24:59.975343 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:24:59.975355 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:24:59.975486 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 20:24:59.975691 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 20:24:59.975766 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 20:24:59.977331 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:59.970773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44071 (local address 127.12.78.3:39853)
0809 20:24:59.971751 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0809 20:24:59.971795 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 20:24:59.973434 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:24:59.974140 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:24:59.974147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:24:59.974236 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:24:59.974956 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:24:59.974959 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:24:59.975000 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:24:59.975086 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:24:59.975451 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:24:59.975469 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:24:59.975478 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 20:24:59.975777 (+ 299us) client_negotiation.cc:240] Negotiation successful
0809 20:24:59.976047 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":294,"threads_started":1}
I20260809 20:24:59.979005 23478 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44071
I20260809 20:24:59.979295 23478 heartbeater.cc:464] Registering TS with master...
I20260809 20:24:59.979879 23478 heartbeater.cc:511] Master 127.12.78.62:44071 requested a full tablet report, sending...
I20260809 20:24:59.981294 22993 ts_manager.cc:194] Registered new tserver with Master: 294b5970e4284d2f87402c724a7e5a6b (127.12.78.3:39255)
I20260809 20:24:59.991241 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 20:24:59.997826 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:24:59.996090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.12.78.62:44071)
0809 20:24:59.996312 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 20:24:59.996320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:24:59.996880 (+ 560us) server_negotiation.cc:407] Connection header received
0809 20:24:59.997031 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:24:59.997035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:24:59.997091 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:24:59.997181 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:24:59.997552 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56284: BlockingRecv error: recv got EOF from 127.0.0.1:56284 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 20:24:59.998473 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 23060
I20260809 20:25:00.002784 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:24:59.999832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.12.78.1:38563)
0809 20:25:00.000477 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:00.000484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:00.000512 (+ 28us) server_negotiation.cc:407] Connection header received
0809 20:25:00.000637 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:00.000648 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:00.000987 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:00.001135 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:00.002105 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:00.002112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:00.002123 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:25:00.002140 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:00.002179 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:25:00.002194 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:00.002301 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:00.002382 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 20:25:00.002509 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":352,"threads_started":1}
I20260809 20:25:00.004155 23186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:25:00.084455 23192 heartbeater.cc:503] Master 127.12.78.62:44071 was elected leader, sending a full tablet report...
I20260809 20:25:00.086998 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 23060
I20260809 20:25:00.102552 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 23196
I20260809 20:25:00.106475 23491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:00.103440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.12.78.2:39239)
0809 20:25:00.104091 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:00.104098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:00.104124 (+ 26us) server_negotiation.cc:407] Connection header received
0809 20:25:00.104214 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:00.104226 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:00.104494 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:00.104736 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:00.105746 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:00.105753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:00.105764 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:25:00.105782 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:00.105820 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:25:00.105834 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:00.105981 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:00.106090 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 20:25:00.106194 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":357,"threads_started":1}
I20260809 20:25:00.107753 23328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:25:00.189606 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 23196
I20260809 20:25:00.208458 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 23338
I20260809 20:25:00.212752 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:00.209605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.12.78.3:39255)
0809 20:25:00.210336 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:00.210343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:00.210375 (+ 32us) server_negotiation.cc:407] Connection header received
0809 20:25:00.210438 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:00.210444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:00.210661 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:00.210788 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:00.211893 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:00.211900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:00.211911 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:25:00.211928 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:00.211968 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:25:00.211983 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:00.212122 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:00.212230 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 20:25:00.212345 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":397,"threads_started":1}
I20260809 20:25:00.214080 23473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:25:00.298308 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 23338
I20260809 20:25:00.318827 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 22960
I20260809 20:25:00.320251 23028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:25:00.424687 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 22960
2026-08-09T20:25:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2451 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:25:00 dist-test-slave-rnrw krb5kdc[23498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:25:00 dist-test-slave-rnrw krb5kdc[23498](info): set up 2 sockets
Aug 09 20:25:00 dist-test-slave-rnrw krb5kdc[23498](info): commencing operation
krb5kdc: starting...
W20260809 20:25:01.589963 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:25:01 dist-test-slave-rnrw krb5kdc[23498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:25:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:25:01.875747 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:42847
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:42847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:25:02.226413 23513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:25:02.226765 23513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:25:02.226864 23513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:25:02.234431 23513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:25:02.234561 23513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:25:02.234611 23513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:25:02.234668 23513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:25:02.234715 23513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:25:02.244994 23513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:42847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:42847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:25:02.246827 23513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:25:02.248293 23513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:25:02.261071 23518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:02.261094 23520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:02.261071 23522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:25:02.271221 23513 server_base.cc:1034] running on GCE node
I20260809 20:25:02.272270 23513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:25:02.274034 23513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:25:02.275310 23513 hybrid_clock.cc:648] HybridClock initialized: now 1786307102275263 us; error 40 us; skew 500 ppm
I20260809 20:25:02.275718 23513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:25:02 dist-test-slave-rnrw krb5kdc[23498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307102, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:25:02.284508 23513 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:25:02.286446 23513 webserver.cc:460] Webserver started at http://127.12.78.62:40401/ using document root <none> and password file <none>
I20260809 20:25:02.287276 23513 fs_manager.cc:359] Metadata directory not provided
I20260809 20:25:02.287461 23513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:25:02.287962 23513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:25:02.291519 23513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "50033e8785b841cc87b6604c1e43ef4d"
format_stamp: "Formatted at 2026-08-09 20:25:02 on dist-test-slave-rnrw"
server_key: "707e8a0c6b9142f2260e19f4f4e380ad"
server_key_iv: "5322becaa221e00acc1a64110fefd187"
server_key_version: "encryptionkey@0"
I20260809 20:25:02.292411 23513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "50033e8785b841cc87b6604c1e43ef4d"
format_stamp: "Formatted at 2026-08-09 20:25:02 on dist-test-slave-rnrw"
server_key: "707e8a0c6b9142f2260e19f4f4e380ad"
server_key_iv: "5322becaa221e00acc1a64110fefd187"
server_key_version: "encryptionkey@0"
I20260809 20:25:02.298223 23513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 20:25:02.302004 23529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:02.303715 23513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:25:02.303882 23513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "50033e8785b841cc87b6604c1e43ef4d"
format_stamp: "Formatted at 2026-08-09 20:25:02 on dist-test-slave-rnrw"
server_key: "707e8a0c6b9142f2260e19f4f4e380ad"
server_key_iv: "5322becaa221e00acc1a64110fefd187"
server_key_version: "encryptionkey@0"
I20260809 20:25:02.304088 23513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:25:02.319535 23513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:25:02.324056 23513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:25:02.324334 23513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:25:02.344370 23513 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:42847
I20260809 20:25:02.344409 23587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:42847 every 8 connection(s)
I20260809 20:25:02.345973 23513 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:25:02.349896 23588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:25:02.354315 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 23513
I20260809 20:25:02.354647 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:25:02.355315 12600 external_mini_cluster.cc:1428] Setting key 5a54a02641bb68d80c2433dedec9aa87
I20260809 20:25:02.363093 23588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d: Bootstrap starting.
I20260809 20:25:02.367074 23588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d: Neither blocks nor log segments found. Creating new log.
I20260809 20:25:02.368570 23588 log.cc:824] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d: Log is configured to *not* fsync() on all Append() calls
Aug 09 20:25:02 dist-test-slave-rnrw krb5kdc[23498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:25:02.372015 23588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d: No bootstrap required, opened a new log
I20260809 20:25:02.383070 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:02.357573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.12.78.62:42847)
0809 20:25:02.358430 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:02.358452 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:02.358520 (+ 68us) server_negotiation.cc:407] Connection header received
0809 20:25:02.361221 (+ 2701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:02.361256 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:02.361823 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:02.362302 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:02.363881 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:02.366021 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:02.367367 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:02.368535 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:02.368770 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:02.373220 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:02.373232 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:02.373253 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:02.373308 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:02.377630 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:02.378510 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:02.378525 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:02.378536 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:02.378780 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:02.379230 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:02.379236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:02.379239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:02.380094 (+ 855us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:02.380464 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:02.380957 (+ 493us) server_negotiation.cc:299] Negotiation successful
0809 20:25:02.381513 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":343,"threads_started":1}
I20260809 20:25:02.383574 23588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50033e8785b841cc87b6604c1e43ef4d" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42847 } }
I20260809 20:25:02.384016 23588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:25:02.384132 23588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50033e8785b841cc87b6604c1e43ef4d, State: Initialized, Role: FOLLOWER
I20260809 20:25:02.384954 23588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50033e8785b841cc87b6604c1e43ef4d" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42847 } }
I20260809 20:25:02.385272 23588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:25:02.385430 23588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:25:02.385617 23588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:25:02.388258 23588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50033e8785b841cc87b6604c1e43ef4d" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42847 } }
I20260809 20:25:02.388962 23588 leader_election.cc:302] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50033e8785b841cc87b6604c1e43ef4d; no voters:
I20260809 20:25:02.389662 23588 leader_election.cc:288] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:25:02.389803 23593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:25:02.390556 23593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [term 1 LEADER]: Becoming Leader. State: Replica: 50033e8785b841cc87b6604c1e43ef4d, State: Running, Role: LEADER
I20260809 20:25:02.391146 23593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50033e8785b841cc87b6604c1e43ef4d" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42847 } }
I20260809 20:25:02.391966 23588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:25:02.393976 23595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50033e8785b841cc87b6604c1e43ef4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50033e8785b841cc87b6604c1e43ef4d" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42847 } } }
I20260809 20:25:02.394385 23595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [sys.catalog]: This master's current role is: LEADER
I20260809 20:25:02.394688 23596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50033e8785b841cc87b6604c1e43ef4d. Latest consensus state: current_term: 1 leader_uuid: "50033e8785b841cc87b6604c1e43ef4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50033e8785b841cc87b6604c1e43ef4d" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42847 } } }
I20260809 20:25:02.395175 23596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d [sys.catalog]: This master's current role is: LEADER
I20260809 20:25:02.409122 23602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:25:02.415428 23602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:25:02.425633 23602 catalog_manager.cc:1347] Generated new cluster ID: 5125719187024ff98392db401cfb02b4
I20260809 20:25:02.425803 23602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:25:02.440689 23602 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:25:02.442111 23602 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:25:02.450285 23602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d: Generated new TSK 0
I20260809 20:25:02.451445 23602 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:25:02.540506 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42847
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:25:02.954344 23617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:25:02.954644 23617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:25:02.954824 23617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:25:02.962178 23617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:25:02.962302 23617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:25:02.962523 23617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:25:02.972726 23617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:25:02.974146 23617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:25:02.975545 23617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:25:02.989161 23622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:02.989127 23625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:02.989207 23623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:25:02.990588 23617 server_base.cc:1034] running on GCE node
I20260809 20:25:02.991326 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:25:02.992437 23617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:25:02.993714 23617 hybrid_clock.cc:648] HybridClock initialized: now 1786307102993652 us; error 57 us; skew 500 ppm
I20260809 20:25:02.994082 23617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:25:02 dist-test-slave-rnrw krb5kdc[23498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307102, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:25:02.997638 23617 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:25:02.999188 23617 webserver.cc:460] Webserver started at http://127.12.78.1:44709/ using document root <none> and password file <none>
I20260809 20:25:03.000003 23617 fs_manager.cc:359] Metadata directory not provided
I20260809 20:25:03.000115 23617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:25:03.000376 23617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:25:03.004127 23617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "0bcc8e1c62e04daba323df35c97fddbe"
format_stamp: "Formatted at 2026-08-09 20:25:02 on dist-test-slave-rnrw"
server_key: "e2d7200de196271afa72127f5fcfc36b"
server_key_iv: "015efb6c77430be67c3a3262577a237e"
server_key_version: "encryptionkey@0"
I20260809 20:25:03.005005 23617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "0bcc8e1c62e04daba323df35c97fddbe"
format_stamp: "Formatted at 2026-08-09 20:25:02 on dist-test-slave-rnrw"
server_key: "e2d7200de196271afa72127f5fcfc36b"
server_key_iv: "015efb6c77430be67c3a3262577a237e"
server_key_version: "encryptionkey@0"
I20260809 20:25:03.011309 23617 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260809 20:25:03.015317 23633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:03.016993 23617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 20:25:03.017167 23617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "0bcc8e1c62e04daba323df35c97fddbe"
format_stamp: "Formatted at 2026-08-09 20:25:02 on dist-test-slave-rnrw"
server_key: "e2d7200de196271afa72127f5fcfc36b"
server_key_iv: "015efb6c77430be67c3a3262577a237e"
server_key_version: "encryptionkey@0"
I20260809 20:25:03.017388 23617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:25:03.047737 23617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:25:03.052232 23617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:25:03.052583 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:25:03.053594 23617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:25:03.055013 23617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:25:03.055087 23617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:03.055245 23617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:25:03.055339 23617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:03.089299 23617 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:33567
I20260809 20:25:03.089398 23756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:33567 every 8 connection(s)
I20260809 20:25:03.090647 23617 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:25:03.091730 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 23617
I20260809 20:25:03.091969 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:25:03.092738 12600 external_mini_cluster.cc:1428] Setting key c8fd0a27cbbc0d30d058385575e5e941
Aug 09 20:25:03 dist-test-slave-rnrw krb5kdc[23498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307102, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:25:03.116078 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:03.093500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:46197 (local address 127.12.78.62:42847)
0809 20:25:03.094056 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:03.094062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:03.095224 (+ 1162us) server_negotiation.cc:407] Connection header received
0809 20:25:03.098213 (+ 2989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:03.098217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:03.098289 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:03.098385 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:03.100609 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:03.101356 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:03.103940 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:03.104364 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:03.104464 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:03.108643 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:03.108653 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:03.108661 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:03.108710 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:03.112429 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:03.113269 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:03.113278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:03.113284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:03.113453 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:03.113915 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:03.113923 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:03.113929 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:03.114359 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:03.114539 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:03.115252 (+ 713us) server_negotiation.cc:299] Negotiation successful
0809 20:25:03.115546 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":832,"threads_started":1}
I20260809 20:25:03.116755 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:03.094202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42847 (local address 127.12.78.1:46197)
0809 20:25:03.095044 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:03.095101 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:03.097893 (+ 2792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:03.098680 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:03.098694 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:03.099492 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:25:03.100348 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:03.100376 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:03.101529 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:03.101537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:03.103778 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:03.103795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:03.104630 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:03.104641 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:03.104981 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:03.105736 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:25:03.105772 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:25:03.108351 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:25:03.112658 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:25:03.112666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:25:03.112681 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:25:03.113127 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:25:03.113568 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:25:03.113571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:25:03.113574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:25:03.113787 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:25:03.114491 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:25:03.114497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:25:03.114847 (+ 350us) client_negotiation.cc:769] Sending connection context
0809 20:25:03.115073 (+ 226us) client_negotiation.cc:240] Negotiation successful
0809 20:25:03.115493 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":363,"threads_started":1}
I20260809 20:25:03.118420 23757 heartbeater.cc:347] Connected to a master server at 127.12.78.62:42847
I20260809 20:25:03.118768 23757 heartbeater.cc:464] Registering TS with master...
I20260809 20:25:03.119771 23757 heartbeater.cc:511] Master 127.12.78.62:42847 requested a full tablet report, sending...
I20260809 20:25:03.122577 23548 ts_manager.cc:194] Registered new tserver with Master: 0bcc8e1c62e04daba323df35c97fddbe (127.12.78.1:33567)
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
I20260809 20:25:03.124126 23548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:46197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:25:03.157925 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42847
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:25:03.510425 23763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:25:03.510718 23763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:25:03.510895 23763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:25:03.518591 23763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:25:03.518705 23763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:25:03.518927 23763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:25:03.529125 23763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:25:03.530476 23763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:25:03.531800 23763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:25:03.545622 23769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:03.545666 23768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:03.546221 23771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:25:03.546489 23763 server_base.cc:1034] running on GCE node
I20260809 20:25:03.547107 23763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:25:03.548086 23763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:25:03.549280 23763 hybrid_clock.cc:648] HybridClock initialized: now 1786307103549237 us; error 34 us; skew 500 ppm
I20260809 20:25:03.549566 23763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:25:03 dist-test-slave-rnrw krb5kdc[23498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307103, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:25:03.552798 23763 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:25:03.554260 23763 webserver.cc:460] Webserver started at http://127.12.78.2:44951/ using document root <none> and password file <none>
I20260809 20:25:03.554917 23763 fs_manager.cc:359] Metadata directory not provided
I20260809 20:25:03.555016 23763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:25:03.555275 23763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:25:03.558218 23763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "431c8592edc549a38932000f942e3074"
format_stamp: "Formatted at 2026-08-09 20:25:03 on dist-test-slave-rnrw"
server_key: "f047d777505d46ffddd71b85af33e5b5"
server_key_iv: "86daf574e75f7bf07de04f60454f10b0"
server_key_version: "encryptionkey@0"
I20260809 20:25:03.558928 23763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "431c8592edc549a38932000f942e3074"
format_stamp: "Formatted at 2026-08-09 20:25:03 on dist-test-slave-rnrw"
server_key: "f047d777505d46ffddd71b85af33e5b5"
server_key_iv: "86daf574e75f7bf07de04f60454f10b0"
server_key_version: "encryptionkey@0"
I20260809 20:25:03.564220 23763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:25:03.567770 23778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:03.569293 23763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:25:03.569469 23763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "431c8592edc549a38932000f942e3074"
format_stamp: "Formatted at 2026-08-09 20:25:03 on dist-test-slave-rnrw"
server_key: "f047d777505d46ffddd71b85af33e5b5"
server_key_iv: "86daf574e75f7bf07de04f60454f10b0"
server_key_version: "encryptionkey@0"
I20260809 20:25:03.569676 23763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:25:03.590137 23763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:25:03.595495 23763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:25:03.595791 23763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:25:03.596887 23763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:25:03.598332 23763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:25:03.598414 23763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:03.598516 23763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:25:03.598567 23763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:03.637759 23763 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:42231
I20260809 20:25:03.637882 23897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:42231 every 8 connection(s)
I20260809 20:25:03.639673 23763 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:25:03.645812 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 23763
I20260809 20:25:03.646045 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:25:03.646595 12600 external_mini_cluster.cc:1428] Setting key da6dfd5d7a776cd5f7fd31af8519cf9f
Aug 09 20:25:03 dist-test-slave-rnrw krb5kdc[23498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307103, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:25:03.668704 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:03.643314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:50993 (local address 127.12.78.62:42847)
0809 20:25:03.643889 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:03.643894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:03.645154 (+ 1260us) server_negotiation.cc:407] Connection header received
0809 20:25:03.647803 (+ 2649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:03.647809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:03.647903 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:03.647985 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:03.650428 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:03.651590 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:03.654377 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:03.655112 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:03.655265 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:03.660408 (+ 5143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:03.660422 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:03.660426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:03.660461 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:03.664299 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:03.665439 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:03.665448 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:03.665454 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:03.665635 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:03.666202 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:03.666210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:03.666216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:03.666674 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:03.666843 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:03.667890 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0809 20:25:03.668157 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":325,"threads_started":1}
I20260809 20:25:03.670293 23901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:03.643897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42847 (local address 127.12.78.2:50993)
0809 20:25:03.644941 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:03.644997 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:03.647470 (+ 2473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:03.648530 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:03.648561 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:03.649373 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:25:03.650219 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:03.650245 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:03.651766 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:03.651775 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:03.654147 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:03.654165 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:03.655388 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:03.655396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:03.655742 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:03.656613 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:25:03.656660 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:25:03.660080 (+ 3420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:25:03.664480 (+ 4400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:25:03.664497 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:25:03.664517 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:25:03.665277 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:25:03.665762 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:25:03.665769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:25:03.665775 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:25:03.666077 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:25:03.666821 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:25:03.666832 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:25:03.667385 (+ 553us) client_negotiation.cc:769] Sending connection context
0809 20:25:03.667768 (+ 383us) client_negotiation.cc:240] Negotiation successful
0809 20:25:03.668441 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":428,"threads_started":1}
I20260809 20:25:03.672039 23898 heartbeater.cc:347] Connected to a master server at 127.12.78.62:42847
I20260809 20:25:03.672397 23898 heartbeater.cc:464] Registering TS with master...
I20260809 20:25:03.673504 23898 heartbeater.cc:511] Master 127.12.78.62:42847 requested a full tablet report, sending...
I20260809 20:25:03.675333 23548 ts_manager.cc:194] Registered new tserver with Master: 431c8592edc549a38932000f942e3074 (127.12.78.2:42231)
I20260809 20:25:03.676721 23548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:50993
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:25:03.712687 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42847
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:25:04.052163 23904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:25:04.052536 23904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:25:04.052747 23904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:25:04.060388 23904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:25:04.060585 23904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:25:04.060781 23904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:25:04.070852 23904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:25:04.072263 23904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:25:04.073822 23904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:25:04.087487 23910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:04.087424 23912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:04.087512 23909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:25:04.088280 23904 server_base.cc:1034] running on GCE node
I20260809 20:25:04.089037 23904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:25:04.090013 23904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:25:04.091255 23904 hybrid_clock.cc:648] HybridClock initialized: now 1786307104091197 us; error 52 us; skew 500 ppm
I20260809 20:25:04.091629 23904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:25:04 dist-test-slave-rnrw krb5kdc[23498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307104, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:25:04.095218 23904 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:25:04.096951 23904 webserver.cc:460] Webserver started at http://127.12.78.3:33471/ using document root <none> and password file <none>
I20260809 20:25:04.097798 23904 fs_manager.cc:359] Metadata directory not provided
I20260809 20:25:04.097903 23904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:25:04.098165 23904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:25:04.101222 23904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "e71fc316dae94cb79b6589719969d5eb"
format_stamp: "Formatted at 2026-08-09 20:25:04 on dist-test-slave-rnrw"
server_key: "5701400993ad86b1bb86bb5f5ec5744d"
server_key_iv: "44dfe01a3933742e8d615c36756b7942"
server_key_version: "encryptionkey@0"
I20260809 20:25:04.101961 23904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "e71fc316dae94cb79b6589719969d5eb"
format_stamp: "Formatted at 2026-08-09 20:25:04 on dist-test-slave-rnrw"
server_key: "5701400993ad86b1bb86bb5f5ec5744d"
server_key_iv: "44dfe01a3933742e8d615c36756b7942"
server_key_version: "encryptionkey@0"
I20260809 20:25:04.107299 23904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:25:04.110978 23919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:04.112310 23904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 20:25:04.112489 23904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "e71fc316dae94cb79b6589719969d5eb"
format_stamp: "Formatted at 2026-08-09 20:25:04 on dist-test-slave-rnrw"
server_key: "5701400993ad86b1bb86bb5f5ec5744d"
server_key_iv: "44dfe01a3933742e8d615c36756b7942"
server_key_version: "encryptionkey@0"
I20260809 20:25:04.112790 23904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:25:04.127369 23757 heartbeater.cc:503] Master 127.12.78.62:42847 was elected leader, sending a full tablet report...
I20260809 20:25:04.145098 23904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:25:04.149844 23904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:25:04.150200 23904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:25:04.151274 23904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:25:04.152750 23904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:25:04.152827 23904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:04.153009 23904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:25:04.153065 23904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:04.188501 23904 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:41477
I20260809 20:25:04.188629 24044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:41477 every 8 connection(s)
I20260809 20:25:04.189919 23904 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:25:04.200009 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 23904
I20260809 20:25:04.200248 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:25:04.200758 12600 external_mini_cluster.cc:1428] Setting key 7d2b6a23b987ac9b91ac917574ef5e67
Aug 09 20:25:04 dist-test-slave-rnrw krb5kdc[23498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307104, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:25:04.210969 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.192693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:55113 (local address 127.12.78.62:42847)
0809 20:25:04.193261 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:04.193266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:04.194123 (+ 857us) server_negotiation.cc:407] Connection header received
0809 20:25:04.196004 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:04.196009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:04.196129 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:04.196254 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:04.197909 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.198714 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.200367 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.200887 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.201033 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.204803 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:04.204808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:04.204811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:04.204845 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:04.207621 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:04.208492 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:04.208500 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:04.208508 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:04.208700 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:04.209149 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:04.209155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:04.209158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:04.209445 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:04.209625 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:04.210340 (+ 715us) server_negotiation.cc:299] Negotiation successful
0809 20:25:04.210583 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":293,"threads_started":1}
I20260809 20:25:04.211935 24048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.193232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42847 (local address 127.12.78.3:55113)
0809 20:25:04.193949 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:04.193995 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:04.195709 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:04.196513 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:04.196522 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:04.197134 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:25:04.197714 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:04.197732 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.198890 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.198900 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:04.200197 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:04.200214 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.201153 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.201161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:04.201405 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.202128 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:25:04.202152 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:25:04.204512 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:25:04.207781 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:25:04.207798 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:25:04.207812 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:25:04.208359 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:25:04.208817 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:25:04.208822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:25:04.208827 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:25:04.209018 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:25:04.209578 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:25:04.209586 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:25:04.209952 (+ 366us) client_negotiation.cc:769] Sending connection context
0809 20:25:04.210241 (+ 289us) client_negotiation.cc:240] Negotiation successful
0809 20:25:04.210802 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":303,"threads_started":1}
I20260809 20:25:04.213480 24045 heartbeater.cc:347] Connected to a master server at 127.12.78.62:42847
I20260809 20:25:04.213831 24045 heartbeater.cc:464] Registering TS with master...
I20260809 20:25:04.214787 24045 heartbeater.cc:511] Master 127.12.78.62:42847 requested a full tablet report, sending...
I20260809 20:25:04.216609 23548 ts_manager.cc:194] Registered new tserver with Master: e71fc316dae94cb79b6589719969d5eb (127.12.78.3:41477)
I20260809 20:25:04.217590 23548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:55113
I20260809 20:25:04.229506 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:25:04.247341 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.234298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.12.78.62:42847)
0809 20:25:04.234482 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:04.234487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:04.234953 (+ 466us) server_negotiation.cc:407] Connection header received
0809 20:25:04.235128 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:04.235132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:04.235213 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:04.235302 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:04.236885 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.237632 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.238906 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.239290 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.239382 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.241828 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:04.241833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:04.241836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:04.241867 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:04.244632 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:04.245369 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:04.245374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:04.245376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:04.245534 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:04.245945 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:04.245949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:04.245951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:04.246354 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:04.246568 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:04.246867 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 20:25:04.247077 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 20:25:04.251848 23548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:25:04.259733 23548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:25:04.286213 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.274894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:33567 (local address 127.0.0.1:43234)
0809 20:25:04.275955 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:04.275976 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:04.276120 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:04.277581 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:04.277586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:04.277620 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:25:04.278218 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:04.278236 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.280455 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.280460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:04.283340 (+ 2880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:04.283359 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.285252 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.285259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:04.285457 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.285502 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 20:25:04.285563 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 20:25:04.285685 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"mutex_wait_us":115,"thread_start_us":744,"threads_started":1}
I20260809 20:25:04.287053 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.274015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.12.78.3:41477)
0809 20:25:04.275606 (+ 1591us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:04.275616 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:04.275657 (+ 41us) server_negotiation.cc:407] Connection header received
0809 20:25:04.275956 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:04.275968 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:04.276312 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:04.276519 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:25:04.278277 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.279839 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.284055 (+ 4216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.285520 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.286310 (+ 790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.286552 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:04.286629 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 20:25:04.286786 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1360,"thread_start_us":1176,"threads_started":1}
I20260809 20:25:04.287070 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.276667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:42231 (local address 127.0.0.1:34690)
0809 20:25:04.277804 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:04.277827 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:04.277941 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:04.279210 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:04.279216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:04.279244 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:25:04.279628 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:04.279639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.281400 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.281406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:04.283253 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:04.283270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.284966 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.284974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:04.285244 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.285311 (+ 67us) client_negotiation.cc:769] Sending connection context
0809 20:25:04.285463 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 20:25:04.285685 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":49,"thread_start_us":859,"threads_started":1}
I20260809 20:25:04.287477 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.275188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.12.78.1:33567)
0809 20:25:04.276846 (+ 1658us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:04.276860 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:04.276899 (+ 39us) server_negotiation.cc:407] Connection header received
0809 20:25:04.277016 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:04.277025 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:04.277297 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:04.278483 (+ 1186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:25:04.278745 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.280300 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.283578 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.284971 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.286547 (+ 1576us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.286816 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:04.286892 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 20:25:04.287082 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1440,"thread_start_us":1218,"threads_started":1}
I20260809 20:25:04.287490 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.277173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.12.78.2:42231)
0809 20:25:04.278656 (+ 1483us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:04.278664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:04.278690 (+ 26us) server_negotiation.cc:407] Connection header received
0809 20:25:04.278791 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:04.278800 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:04.278996 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:04.279193 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:25:04.279819 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.281216 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.283421 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.284739 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.284875 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.285091 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:04.285827 (+ 736us) server_negotiation.cc:299] Negotiation successful
0809 20:25:04.286039 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1325,"thread_start_us":1217,"threads_started":1}
I20260809 20:25:04.287636 24059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.273845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:41477 (local address 127.0.0.1:51392)
0809 20:25:04.274662 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:04.274710 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:04.274880 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:04.276747 (+ 1867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:04.276754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:04.276788 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:25:04.277233 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:04.277249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.279994 (+ 2745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.280000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:04.283863 (+ 3863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:04.283878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.285760 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.285768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:04.285929 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.285958 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 20:25:04.286025 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 20:25:04.286187 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":973,"thread_start_us":401,"threads_started":1}
I20260809 20:25:04.291613 23688 tablet_service.cc:1467] Processing CreateTablet for tablet a6626dc99025489bafda7763ca5e7802 (DEFAULT_TABLE table=test-table [id=503897c9e08e456db0f0368c8f143210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:25:04.291646 23832 tablet_service.cc:1467] Processing CreateTablet for tablet a6626dc99025489bafda7763ca5e7802 (DEFAULT_TABLE table=test-table [id=503897c9e08e456db0f0368c8f143210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:25:04.293165 23688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6626dc99025489bafda7763ca5e7802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:25:04.293174 23832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6626dc99025489bafda7763ca5e7802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:25:04.293540 23973 tablet_service.cc:1467] Processing CreateTablet for tablet a6626dc99025489bafda7763ca5e7802 (DEFAULT_TABLE table=test-table [id=503897c9e08e456db0f0368c8f143210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:25:04.294667 23973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6626dc99025489bafda7763ca5e7802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:25:04.306195 24065 tablet_bootstrap.cc:492] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb: Bootstrap starting.
I20260809 20:25:04.308507 24066 tablet_bootstrap.cc:492] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074: Bootstrap starting.
I20260809 20:25:04.310577 24065 tablet_bootstrap.cc:654] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb: Neither blocks nor log segments found. Creating new log.
I20260809 20:25:04.310554 24067 tablet_bootstrap.cc:492] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe: Bootstrap starting.
I20260809 20:25:04.311993 24065 log.cc:824] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb: Log is configured to *not* fsync() on all Append() calls
I20260809 20:25:04.314492 24067 tablet_bootstrap.cc:654] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe: Neither blocks nor log segments found. Creating new log.
I20260809 20:25:04.315550 24067 log.cc:824] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe: Log is configured to *not* fsync() on all Append() calls
I20260809 20:25:04.315656 24065 tablet_bootstrap.cc:492] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb: No bootstrap required, opened a new log
I20260809 20:25:04.315981 24066 tablet_bootstrap.cc:654] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074: Neither blocks nor log segments found. Creating new log.
I20260809 20:25:04.316026 24065 ts_tablet_manager.cc:1397] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260809 20:25:04.317291 24066 log.cc:824] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074: Log is configured to *not* fsync() on all Append() calls
I20260809 20:25:04.319077 24067 tablet_bootstrap.cc:492] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe: No bootstrap required, opened a new log
I20260809 20:25:04.319355 24067 ts_tablet_manager.cc:1397] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 20:25:04.320474 24066 tablet_bootstrap.cc:492] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074: No bootstrap required, opened a new log
I20260809 20:25:04.320770 24066 ts_tablet_manager.cc:1397] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20260809 20:25:04.326726 24065 raft_consensus.cc:348] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e71fc316dae94cb79b6589719969d5eb" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41477 } } peers { permanent_uuid: "0bcc8e1c62e04daba323df35c97fddbe" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33567 } } peers { permanent_uuid: "431c8592edc549a38932000f942e3074" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42231 } }
I20260809 20:25:04.327237 24065 raft_consensus.cc:374] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:25:04.327399 24065 raft_consensus.cc:729] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e71fc316dae94cb79b6589719969d5eb, State: Initialized, Role: FOLLOWER
I20260809 20:25:04.327456 24067 raft_consensus.cc:348] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e71fc316dae94cb79b6589719969d5eb" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41477 } } peers { permanent_uuid: "0bcc8e1c62e04daba323df35c97fddbe" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33567 } } peers { permanent_uuid: "431c8592edc549a38932000f942e3074" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42231 } }
I20260809 20:25:04.327907 24067 raft_consensus.cc:374] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:25:04.328035 24067 raft_consensus.cc:729] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcc8e1c62e04daba323df35c97fddbe, State: Initialized, Role: FOLLOWER
I20260809 20:25:04.328078 24065 consensus_queue.cc:260] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e71fc316dae94cb79b6589719969d5eb" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41477 } } peers { permanent_uuid: "0bcc8e1c62e04daba323df35c97fddbe" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33567 } } peers { permanent_uuid: "431c8592edc549a38932000f942e3074" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42231 } }
I20260809 20:25:04.328679 24067 consensus_queue.cc:260] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e71fc316dae94cb79b6589719969d5eb" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41477 } } peers { permanent_uuid: "0bcc8e1c62e04daba323df35c97fddbe" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33567 } } peers { permanent_uuid: "431c8592edc549a38932000f942e3074" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42231 } }
I20260809 20:25:04.330585 24067 ts_tablet_manager.cc:1428] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe: Time spent starting tablet: real 0.011s user 0.008s sys 0.005s
I20260809 20:25:04.332006 24045 heartbeater.cc:503] Master 127.12.78.62:42847 was elected leader, sending a full tablet report...
I20260809 20:25:04.332280 24066 raft_consensus.cc:348] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e71fc316dae94cb79b6589719969d5eb" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41477 } } peers { permanent_uuid: "0bcc8e1c62e04daba323df35c97fddbe" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33567 } } peers { permanent_uuid: "431c8592edc549a38932000f942e3074" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42231 } }
I20260809 20:25:04.332777 24066 raft_consensus.cc:374] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:25:04.332824 24065 ts_tablet_manager.cc:1428] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260809 20:25:04.332880 24066 raft_consensus.cc:729] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431c8592edc549a38932000f942e3074, State: Initialized, Role: FOLLOWER
I20260809 20:25:04.333524 24066 consensus_queue.cc:260] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e71fc316dae94cb79b6589719969d5eb" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41477 } } peers { permanent_uuid: "0bcc8e1c62e04daba323df35c97fddbe" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33567 } } peers { permanent_uuid: "431c8592edc549a38932000f942e3074" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42231 } }
I20260809 20:25:04.334756 23898 heartbeater.cc:503] Master 127.12.78.62:42847 was elected leader, sending a full tablet report...
I20260809 20:25:04.336004 24066 ts_tablet_manager.cc:1428] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260809 20:25:04.343161 23758 tablet.cc:2369] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:25:04.392098 23899 tablet.cc:2369] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:25:04.441681 24046 tablet.cc:2369] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:25:04.593624 24073 raft_consensus.cc:482] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:25:04.594066 24073 raft_consensus.cc:504] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e71fc316dae94cb79b6589719969d5eb" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41477 } } peers { permanent_uuid: "0bcc8e1c62e04daba323df35c97fddbe" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33567 } } peers { permanent_uuid: "431c8592edc549a38932000f942e3074" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42231 } }
I20260809 20:25:04.596266 24073 leader_election.cc:288] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e71fc316dae94cb79b6589719969d5eb (127.12.78.3:41477), 0bcc8e1c62e04daba323df35c97fddbe (127.12.78.1:33567)
I20260809 20:25:04.603309 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.596931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:39103 (local address 127.12.78.3:41477)
0809 20:25:04.597159 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:04.597166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:04.597529 (+ 363us) server_negotiation.cc:407] Connection header received
0809 20:25:04.597690 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:04.597697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:04.597811 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:04.597961 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:25:04.598649 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.599614 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.601524 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.602307 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.602471 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.602592 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:04.602895 (+ 303us) server_negotiation.cc:299] Negotiation successful
0809 20:25:04.603038 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 20:25:04.603421 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.596638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:41477 (local address 127.12.78.2:39103)
0809 20:25:04.597394 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:04.597414 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:04.597555 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:04.597966 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:04.597969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:04.597991 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:25:04.598427 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:04.598478 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.599750 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.599758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:04.601336 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:04.601350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.602577 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.602584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:04.602724 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.602747 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 20:25:04.602864 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 20:25:04.603005 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":4,"thread_start_us":374,"threads_started":1}
I20260809 20:25:04.604503 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.597462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:33567 (local address 127.12.78.2:33663)
0809 20:25:04.598763 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:04.598779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:04.598887 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:04.599208 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:04.599212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:04.599231 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:25:04.599625 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:04.599635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.600851 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.600857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:04.602392 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:04.602406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.603824 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.603831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:04.604017 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.604061 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 20:25:04.604160 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 20:25:04.604253 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"thread_start_us":915,"threads_started":1}
I20260809 20:25:04.604712 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.596923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:33663 (local address 127.12.78.1:33567)
0809 20:25:04.597158 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:04.597165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:04.598881 (+ 1716us) server_negotiation.cc:407] Connection header received
0809 20:25:04.598998 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:04.599003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:04.599085 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:04.599185 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:25:04.599808 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.600697 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.602576 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.603590 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.603736 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.603847 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:04.604227 (+ 380us) server_negotiation.cc:299] Negotiation successful
0809 20:25:04.604392 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 20:25:04.604857 23993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6626dc99025489bafda7763ca5e7802" candidate_uuid: "431c8592edc549a38932000f942e3074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e71fc316dae94cb79b6589719969d5eb" is_pre_election: true
I20260809 20:25:04.605459 23993 raft_consensus.cc:2454] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 431c8592edc549a38932000f942e3074 in term 0.
I20260809 20:25:04.605829 23709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6626dc99025489bafda7763ca5e7802" candidate_uuid: "431c8592edc549a38932000f942e3074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bcc8e1c62e04daba323df35c97fddbe" is_pre_election: true
I20260809 20:25:04.606268 23781 leader_election.cc:302] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 431c8592edc549a38932000f942e3074, e71fc316dae94cb79b6589719969d5eb; no voters:
I20260809 20:25:04.606458 23709 raft_consensus.cc:2454] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 431c8592edc549a38932000f942e3074 in term 0.
I20260809 20:25:04.606709 24073 raft_consensus.cc:2781] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:25:04.606863 24073 raft_consensus.cc:482] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:25:04.607074 24073 raft_consensus.cc:3037] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:25:04.609853 24073 raft_consensus.cc:504] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e71fc316dae94cb79b6589719969d5eb" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41477 } } peers { permanent_uuid: "0bcc8e1c62e04daba323df35c97fddbe" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33567 } } peers { permanent_uuid: "431c8592edc549a38932000f942e3074" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42231 } }
I20260809 20:25:04.610514 24073 leader_election.cc:288] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [CANDIDATE]: Term 1 election: Requested vote from peers e71fc316dae94cb79b6589719969d5eb (127.12.78.3:41477), 0bcc8e1c62e04daba323df35c97fddbe (127.12.78.1:33567)
I20260809 20:25:04.611001 23993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6626dc99025489bafda7763ca5e7802" candidate_uuid: "431c8592edc549a38932000f942e3074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e71fc316dae94cb79b6589719969d5eb"
I20260809 20:25:04.611127 23709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6626dc99025489bafda7763ca5e7802" candidate_uuid: "431c8592edc549a38932000f942e3074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bcc8e1c62e04daba323df35c97fddbe"
I20260809 20:25:04.611235 23993 raft_consensus.cc:3037] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:25:04.611388 23709 raft_consensus.cc:3037] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:25:04.613936 23709 raft_consensus.cc:2454] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 431c8592edc549a38932000f942e3074 in term 1.
I20260809 20:25:04.613946 23993 raft_consensus.cc:2454] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 431c8592edc549a38932000f942e3074 in term 1.
I20260809 20:25:04.614437 23780 leader_election.cc:302] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bcc8e1c62e04daba323df35c97fddbe, 431c8592edc549a38932000f942e3074; no voters:
I20260809 20:25:04.614781 24073 raft_consensus.cc:2781] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:25:04.615265 24073 raft_consensus.cc:686] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [term 1 LEADER]: Becoming Leader. State: Replica: 431c8592edc549a38932000f942e3074, State: Running, Role: LEADER
I20260809 20:25:04.615818 24073 consensus_queue.cc:237] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e71fc316dae94cb79b6589719969d5eb" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41477 } } peers { permanent_uuid: "0bcc8e1c62e04daba323df35c97fddbe" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33567 } } peers { permanent_uuid: "431c8592edc549a38932000f942e3074" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42231 } }
I20260809 20:25:04.621209 23548 catalog_manager.cc:5495] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 reported cstate change: term changed from 0 to 1, leader changed from <none> to 431c8592edc549a38932000f942e3074 (127.12.78.2). New cstate: current_term: 1 leader_uuid: "431c8592edc549a38932000f942e3074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e71fc316dae94cb79b6589719969d5eb" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bcc8e1c62e04daba323df35c97fddbe" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "431c8592edc549a38932000f942e3074" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42231 } health_report { overall_health: HEALTHY } } }
I20260809 20:25:04.653396 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.647993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.12.78.2:42231)
0809 20:25:04.648241 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:04.648248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:04.648279 (+ 31us) server_negotiation.cc:407] Connection header received
0809 20:25:04.648477 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:04.648481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:04.648634 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:04.648763 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:25:04.649091 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.649805 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.651107 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.651523 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.651622 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.652132 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:25:04.652747 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:25:04.652864 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:04.652979 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 20:25:04.653059 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 20:25:04.662621 23709 raft_consensus.cc:1261] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe [term 1 FOLLOWER]: Refusing update from remote peer 431c8592edc549a38932000f942e3074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:25:04.662650 23993 raft_consensus.cc:1261] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb [term 1 FOLLOWER]: Refusing update from remote peer 431c8592edc549a38932000f942e3074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:25:04.663926 24076 consensus_queue.cc:1035] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bcc8e1c62e04daba323df35c97fddbe" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:25:04.664324 24073 consensus_queue.cc:1035] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e71fc316dae94cb79b6589719969d5eb" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:25:04.692687 23548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:25:04.693045 23548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:25:04.696601 23548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50033e8785b841cc87b6604c1e43ef4d: Sending DeleteTablet for 3 replicas of tablet a6626dc99025489bafda7763ca5e7802
I20260809 20:25:04.697773 23973 tablet_service.cc:1514] Processing DeleteTablet for tablet a6626dc99025489bafda7763ca5e7802 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:25:04 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:51392
I20260809 20:25:04.698242 23832 tablet_service.cc:1514] Processing DeleteTablet for tablet a6626dc99025489bafda7763ca5e7802 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:25:04 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:34690
I20260809 20:25:04.698578 23688 tablet_service.cc:1514] Processing DeleteTablet for tablet a6626dc99025489bafda7763ca5e7802 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:25:04 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:43234
I20260809 20:25:04.699307 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 23617
I20260809 20:25:04.713513 24092 tablet_replica.cc:330] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe: stopping tablet replica
I20260809 20:25:04.714875 24092 raft_consensus.cc:2229] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:25:04.715698 24092 raft_consensus.cc:2258] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:25:04.718762 24092 ts_tablet_manager.cc:1905] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:25:04.719707 24090 tablet_replica.cc:330] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb: stopping tablet replica
I20260809 20:25:04.723052 24090 raft_consensus.cc:2229] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:25:04.723728 24090 raft_consensus.cc:2258] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:25:04.726558 24092 ts_tablet_manager.cc:1918] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:25:04.727233 24090 ts_tablet_manager.cc:1905] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:25:04.727280 24091 tablet_replica.cc:330] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074: stopping tablet replica
I20260809 20:25:04.727857 24091 raft_consensus.cc:2229] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:25:04.728657 24091 raft_consensus.cc:2258] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:25:04.729030 24092 log.cc:1196] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/a6626dc99025489bafda7763ca5e7802
I20260809 20:25:04.731331 24091 ts_tablet_manager.cc:1905] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:25:04.732492 24092 ts_tablet_manager.cc:1939] T a6626dc99025489bafda7763ca5e7802 P 0bcc8e1c62e04daba323df35c97fddbe: Deleting consensus metadata
Aug 09 20:25:04 dist-test-slave-rnrw krb5kdc[23498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:25:04.734676 23532 catalog_manager.cc:4841] TS 0bcc8e1c62e04daba323df35c97fddbe (127.12.78.1:33567): tablet a6626dc99025489bafda7763ca5e7802 (table test-table [id=503897c9e08e456db0f0368c8f143210]) successfully deleted
I20260809 20:25:04.735486 24090 ts_tablet_manager.cc:1918] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:25:04.735726 24090 log.cc:1196] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/a6626dc99025489bafda7763ca5e7802
I20260809 20:25:04.736368 24090 ts_tablet_manager.cc:1939] T a6626dc99025489bafda7763ca5e7802 P e71fc316dae94cb79b6589719969d5eb: Deleting consensus metadata
I20260809 20:25:04.738395 23533 catalog_manager.cc:4841] TS e71fc316dae94cb79b6589719969d5eb (127.12.78.3:41477): tablet a6626dc99025489bafda7763ca5e7802 (table test-table [id=503897c9e08e456db0f0368c8f143210]) successfully deleted
I20260809 20:25:04.741501 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.700153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.12.78.1:33567)
0809 20:25:04.700337 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:04.700344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:04.714107 (+ 13763us) server_negotiation.cc:407] Connection header received
0809 20:25:04.715273 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:04.715285 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:04.715395 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:04.715495 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:04.720032 (+ 4537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.721301 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.727133 (+ 5832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.727830 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.727941 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.733901 (+ 5960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:04.733916 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:04.733935 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:04.733980 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:04.738156 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:04.739138 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:04.739147 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:04.739149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:04.739291 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:04.739728 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:04.739738 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:04.739740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:04.740120 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:04.740443 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:04.740852 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 20:25:04.741081 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 20:25:04.741673 24091 ts_tablet_manager.cc:1918] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:25:04.742230 23752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:25:04.742244 24091 log.cc:1196] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/a6626dc99025489bafda7763ca5e7802
I20260809 20:25:04.742833 24091 ts_tablet_manager.cc:1939] T a6626dc99025489bafda7763ca5e7802 P 431c8592edc549a38932000f942e3074: Deleting consensus metadata
I20260809 20:25:04.744801 23532 catalog_manager.cc:4841] TS 431c8592edc549a38932000f942e3074 (127.12.78.2:42231): tablet a6626dc99025489bafda7763ca5e7802 (table test-table [id=503897c9e08e456db0f0368c8f143210]) successfully deleted
I20260809 20:25:04.839851 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 23617
I20260809 20:25:04.866276 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 23763
Aug 09 20:25:04 dist-test-slave-rnrw krb5kdc[23498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:25:04.884369 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:04.867130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.12.78.2:42231)
0809 20:25:04.867339 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:04.867345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:04.867373 (+ 28us) server_negotiation.cc:407] Connection header received
0809 20:25:04.867484 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:04.867492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:04.867591 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:04.867714 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:04.869121 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.870364 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.871668 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:04.872343 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:04.872504 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:04.876997 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:04.877019 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:04.877043 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:04.877096 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:04.881265 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:04.882031 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:04.882041 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:04.882044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:04.882219 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:04.882585 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:04.882591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:04.882594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:04.883137 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:04.883515 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:04.883711 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 20:25:04.883956 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 20:25:04.885277 23893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:25:05.000275 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 23763
I20260809 20:25:05.021946 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 23904
Aug 09 20:25:05 dist-test-slave-rnrw krb5kdc[23498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:25:05.037858 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:05.022712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.12.78.3:41477)
0809 20:25:05.022890 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:05.022895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:05.022917 (+ 22us) server_negotiation.cc:407] Connection header received
0809 20:25:05.023106 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:05.023110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:05.023178 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:05.023266 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:05.024698 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:05.025455 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:05.026776 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:05.027255 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:05.027375 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:05.031951 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:05.031972 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:05.031990 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:05.032034 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:05.035063 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:05.035785 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:05.035792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:05.035794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:05.035948 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:05.036296 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:05.036299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:05.036301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:05.036752 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:05.036996 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:05.037268 (+ 272us) server_negotiation.cc:299] Negotiation successful
0809 20:25:05.037441 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 20:25:05.038533 24040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:25:05.128881 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 23904
I20260809 20:25:05.148066 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 23513
I20260809 20:25:05.149356 23583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:25:05.262720 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 23513
2026-08-09T20:25:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4840 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:25:05 dist-test-slave-rnrw krb5kdc[24100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:25:05 dist-test-slave-rnrw krb5kdc[24100](info): set up 2 sockets
Aug 09 20:25:05 dist-test-slave-rnrw krb5kdc[24100](info): commencing operation
krb5kdc: starting...
W20260809 20:25:06.426898 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:25:06 dist-test-slave-rnrw krb5kdc[24100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:25:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:25:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.78.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.78.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/rangeradmin_127.12.78.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/rangeradmin_127.12.78.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.78.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.78.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/rangerlookup_127.12.78.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/rangerlookup_127.12.78.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.78.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.78.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/HTTP_127.12.78.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/HTTP_127.12.78.20@KRBTEST.COM.keytab.
I20260809 20:25:06.839193 12600 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktNpCrv/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/postgres -l logfile start
2026-08-09 20:25:09.457 UTC [24131] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 20:25:09.457 UTC [24131] LOG: listening on IPv4 address "127.12.78.20", port 50589
2026-08-09 20:25:09.461 UTC [24131] LOG: listening on Unix socket "/tmp/.s.PGSQL.50589"
2026-08-09 20:25:09.486 UTC [24134] LOG: database system was shut down at 2026-08-09 20:25:07 UTC
2026-08-09 20:25:09.490 UTC [24131] LOG: database system is ready to accept connections
I20260809 20:25:10.536980 12600 mini_postgres.cc:96] Postgres bound to 50589
2026-08-09 20:25:10.560 UTC [24142] FATAL: database "slave" does not exist
127.12.78.20:50589 - accepting connections
I20260809 20:25:10.561661 12600 mini_ranger.cc:161] Starting Ranger...
I20260809 20:25:10.595256 12600 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 20:25:11.121546 12600 mini_ranger.cc:87] Created ranger Postgres database
I20260809 20:25:11.121716 12600 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-admin
2026-08-09 20:25:11,540 [I] DB FLAVOR :POSTGRES
2026-08-09 20:25:11,541 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 20:25:11,541 [I] Checking connection..
2026-08-09 20:25:11,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 20:25:11,826 [I] Checking connection passed.
2026-08-09 20:25:11,827 [I] --------- Verifying version history table ---------
2026-08-09 20:25:11,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 20:25:12,104 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 20:25:12,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 20:25:12,377 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 20:25:12,377 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 20:25:12,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 20:25:12.634 UTC [24211] WARNING: there is no transaction in progress
2026-08-09 20:25:12,642 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 20:25:12,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 20:25:12,886 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 20:25:12,887 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 20:25:12,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 20:25:13,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 20:25:13,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rnrw.c.gcp-upstream.internal','N') ;"
2026-08-09 20:25:13,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 20:25:13,936 [I] Table x_portal_user does not exist in database ranger
2026-08-09 20:25:13,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 20:25:14,201 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 20:25:14,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 20:25:14,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 20:25:14,727 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 20:25:14,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 20:25:15.775 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:15.793 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:15.812 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:15.830 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:15.851 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:15.876 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:15.953 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:15.960 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:15.977 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:15.989 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:16.001 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:16.017 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:16.029 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:16.037 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:16.045 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:16.053 UTC [24327] WARNING: there is no transaction in progress
2026-08-09 20:25:16,809 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 20:25:16,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 20:25:17,080 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 20:25:17,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 20:25:17,329 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 20:25:17,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 20:25:17,610 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 20:25:17,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 20:25:17,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 20:25:18,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rnrw.c.gcp-upstream.internal';"
2026-08-09 20:25:18,382 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 20:25:18,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 20:25:18,631 [I] DB_PATCHES have already been applied
I20260809 20:25:18.637928 12600 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktNpCrv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 20:25:18.638032 12600 mini_ranger.cc:193] Using host: 127.12.78.20
I20260809 20:25:18.658229 12600 mini_ranger.cc:227] Ranger admin URL: http://127.12.78.20:52465
Aug 09, 2026 8:25:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 8:25:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 8:25:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 8:25:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 8:25:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 8:25:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 8:25:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 20:25:19.849351 12600 mini_ranger.cc:160] Time spent starting Ranger: real 9.288s user 0.000s sys 0.060s
Aug 09, 2026 8:25:42 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 8:25:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 09, 2026 8:25:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 8:25:42 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 8:25:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 8:25:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 8:25:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 8:25:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 8:25:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 8:25:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 8:25:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 8:25:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 8:25:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 8:25:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 8:25:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 8:25:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 8:25:42 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 8:25:43 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 20:25:43.802767 12600 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.78.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.78.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/rangerkms_127.12.78.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/rangerkms_127.12.78.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.78.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.78.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.78.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/HTTP_127.12.78.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.78.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/HTTP_127.12.78.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 20:25:44 dist-test-slave-rnrw krb5kdc[24100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307144, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 20:25:44.015561 12600 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 20:25:44.042088 12600 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 20:25:44.907287 12600 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 20:25:44.907485 12600 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-kms
I20260809 20:25:44.907610 12600 mini_ranger_kms.cc:227] Using postgres at 127.12.78.20:50589
2026-08-09 20:25:45,478 [I] DB FLAVOR :POSTGRES
2026-08-09 20:25:45,479 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 20:25:45,479 [I] Checking connection
2026-08-09 20:25:45,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 20:25:45,746 [I] connection success
2026-08-09 20:25:45,746 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 20:25:45,746 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 20:25:45,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 20:25:45,998 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 20:25:45,999 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 20:25:45,999 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 20:25:45,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:50589/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 20:25:46,269 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 20:25:46.773298 12600 mini_ranger_kms.cc:310] Created kms service
I20260809 20:25:46.996421 12600 mini_ranger_kms.cc:326] Created kudu user
I20260809 20:25:47.051724 12600 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 20:25:47 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307144, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:25:47.489156 12600 mini_ranger_kms.cc:384] Added ranger policy
I20260809 20:25:47.490971 12600 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 20:25:47.491072 12600 mini_ranger_kms.cc:240] Using host: 127.12.78.20
I20260809 20:25:47.522104 12600 mini_ranger_kms.cc:276] Ranger KMS PID: 24527
I20260809 20:25:47.522346 12600 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.78.20:44955
Aug 09, 2026 8:25:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 8:25:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 8:25:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 8:25:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 8:25:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 8:25:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260809 20:25:49.960276 12600 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.945s user 0.114s sys 0.481s
I20260809 20:25:49.960686 12600 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 20:25:49.960927 12600 mini_ranger_kms.cc:401] 127.12.78.20:44955/kms/v1/keys
Aug 09 20:25:50 dist-test-slave-rnrw krb5kdc[24100](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786307150, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.78.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 09 20:25:50 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307150, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.78.20@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
Aug 09 20:25:50 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307150, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.78.20@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
Aug 09 20:25:53 dist-test-slave-rnrw krb5kdc[24100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:25:53.416532 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44883
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:44883
--ranger_config_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.78.20:44955/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:25:53.925640 24580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:25:53.926141 24580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:25:53.926380 24580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:25:53.935982 24580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:25:53.936138 24580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:25:53.936246 24580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:25:53.936358 24580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:25:53.936486 24580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:25:53.958328 24580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.78.20:44955/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:44883
--ranger_config_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:25:53.960319 24580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:25:53.962446 24580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:25:53.981160 24589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:53.981293 24586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:53.981287 24587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:25:53.984681 24580 server_base.cc:1034] running on GCE node
I20260809 20:25:53.985656 24580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:25:53.987361 24580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:25:53.988687 24580 hybrid_clock.cc:648] HybridClock initialized: now 1786307153988606 us; error 70 us; skew 500 ppm
I20260809 20:25:53.989091 24580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:25:53 dist-test-slave-rnrw krb5kdc[24100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307153, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:25:53.993077 24580 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:25:53.995260 24580 webserver.cc:460] Webserver started at http://127.12.78.62:46809/ using document root <none> and password file <none>
I20260809 20:25:53.996078 24580 fs_manager.cc:359] Metadata directory not provided
I20260809 20:25:53.996245 24580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:25:53.996654 24580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 20:25:53 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307153, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:25:54.062819 24580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "3b71ba63417e463296dfad38b5b6da40"
format_stamp: "Formatted at 2026-08-09 20:25:54 on dist-test-slave-rnrw"
server_key: "0c590a31467a8d52839e18c6bb55edc2"
server_key_iv: "4fa146563826b55c2ca0ba6da87e8d28"
server_key_version: "kuduclusterkey@0"
I20260809 20:25:54.064333 24580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "3b71ba63417e463296dfad38b5b6da40"
format_stamp: "Formatted at 2026-08-09 20:25:54 on dist-test-slave-rnrw"
server_key: "0c590a31467a8d52839e18c6bb55edc2"
server_key_iv: "4fa146563826b55c2ca0ba6da87e8d28"
server_key_version: "kuduclusterkey@0"
I20260809 20:25:54.073431 24580 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260809 20:25:54.093727 24597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:54.095423 24580 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 20:25:54.095695 24580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "3b71ba63417e463296dfad38b5b6da40"
format_stamp: "Formatted at 2026-08-09 20:25:54 on dist-test-slave-rnrw"
server_key: "0c590a31467a8d52839e18c6bb55edc2"
server_key_iv: "4fa146563826b55c2ca0ba6da87e8d28"
server_key_version: "kuduclusterkey@0"
I20260809 20:25:54.096077 24580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:25:54.120956 24580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:25:54.126730 24580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:25:54.127276 24580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:25:54.165706 24580 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:44883
I20260809 20:25:54.165741 24651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:44883 every 8 connection(s)
I20260809 20:25:54.167338 24580 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:25:54.167964 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 24580
I20260809 20:25:54.168193 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:25:54.169204 12600 external_mini_cluster.cc:1428] Setting key 2673201b6c50a778a9b432ec917fc7e8
I20260809 20:25:54.173584 24652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 20:25:54 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:25:54.187897 24652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40: Bootstrap starting.
I20260809 20:25:54.193436 24652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40: Neither blocks nor log segments found. Creating new log.
I20260809 20:25:54.195333 24652 log.cc:824] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40: Log is configured to *not* fsync() on all Append() calls
I20260809 20:25:54.197701 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:54.171708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.12.78.62:44883)
0809 20:25:54.173178 (+ 1470us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:54.173190 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:54.173233 (+ 43us) server_negotiation.cc:407] Connection header received
0809 20:25:54.175431 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:54.175455 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:54.175887 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:54.176408 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:54.177931 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:54.179713 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:54.181104 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:54.181946 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:54.182166 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:54.186746 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:54.186759 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:54.186782 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:54.186835 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:54.192350 (+ 5515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:54.193297 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:54.193309 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:54.193318 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:54.193574 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:54.194046 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:54.194058 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:54.194064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:54.195022 (+ 958us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:54.195412 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:54.195949 (+ 537us) server_negotiation.cc:299] Negotiation successful
0809 20:25:54.196464 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1194,"thread_start_us":854,"threads_started":1}
I20260809 20:25:54.199553 24652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40: No bootstrap required, opened a new log
I20260809 20:25:54.209404 24652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b71ba63417e463296dfad38b5b6da40" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44883 } }
I20260809 20:25:54.209829 24652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:25:54.209944 24652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b71ba63417e463296dfad38b5b6da40, State: Initialized, Role: FOLLOWER
I20260809 20:25:54.210655 24652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b71ba63417e463296dfad38b5b6da40" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44883 } }
I20260809 20:25:54.210883 24652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:25:54.211050 24652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:25:54.211297 24652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:25:54.214509 24652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b71ba63417e463296dfad38b5b6da40" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44883 } }
I20260809 20:25:54.215060 24652 leader_election.cc:302] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b71ba63417e463296dfad38b5b6da40; no voters:
I20260809 20:25:54.215651 24652 leader_election.cc:288] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:25:54.215821 24657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:25:54.217006 24657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [term 1 LEADER]: Becoming Leader. State: Replica: 3b71ba63417e463296dfad38b5b6da40, State: Running, Role: LEADER
I20260809 20:25:54.217639 24657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b71ba63417e463296dfad38b5b6da40" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44883 } }
I20260809 20:25:54.218583 24652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:25:54.221385 24658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b71ba63417e463296dfad38b5b6da40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b71ba63417e463296dfad38b5b6da40" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44883 } } }
I20260809 20:25:54.222293 24658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [sys.catalog]: This master's current role is: LEADER
I20260809 20:25:54.221339 24659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b71ba63417e463296dfad38b5b6da40. Latest consensus state: current_term: 1 leader_uuid: "3b71ba63417e463296dfad38b5b6da40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b71ba63417e463296dfad38b5b6da40" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44883 } } }
I20260809 20:25:54.222909 24659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40 [sys.catalog]: This master's current role is: LEADER
I20260809 20:25:54.226145 24664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:25:54.234661 24664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:25:54.236048 24652 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 20:25:54.257690 24664 catalog_manager.cc:1347] Generated new cluster ID: 677a024ab4084f27ae3b36ef09f61bd4
I20260809 20:25:54.257856 24664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:25:54.292128 24664 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:25:54.294880 24664 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:25:54.319881 24664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40: Generated new TSK 0
I20260809 20:25:54.321089 24664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:25:55.165617 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.78.20:44955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44883
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 20:25:55 dist-test-slave-rnrw krb5kdc[24100](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786307155, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 20:25:55.754796 24698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:25:55.755277 24698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:25:55.755496 24698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:25:55.764729 24698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:25:55.764963 24698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:25:55.765196 24698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:25:55.777674 24698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.78.20:44955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:25:55.779086 24698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:25:55.780591 24698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:25:55.805092 24709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:55.805373 24710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:25:55.811694 24698 server_base.cc:1034] running on GCE node
W20260809 20:25:55.812021 24712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:25:55.813122 24698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:25:55.814540 24698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:25:55.815865 24698 hybrid_clock.cc:648] HybridClock initialized: now 1786307155815757 us; error 97 us; skew 500 ppm
I20260809 20:25:55.816311 24698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:25:55 dist-test-slave-rnrw krb5kdc[24100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307155, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:25:55.821475 24698 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:25:55.836284 24698 webserver.cc:460] Webserver started at http://127.12.78.1:42271/ using document root <none> and password file <none>
I20260809 20:25:55.837708 24698 fs_manager.cc:359] Metadata directory not provided
I20260809 20:25:55.838140 24698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:25:55.838728 24698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 20:25:55 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307155, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:25:55.862915 24698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "5261c7f9c9e04e7fa06370f708feb55b"
format_stamp: "Formatted at 2026-08-09 20:25:55 on dist-test-slave-rnrw"
server_key: "03e6a32863fe1b13eabd4819f454eb0a"
server_key_iv: "1c2ce55b7c6ee2e053bd9b1dcc74d2e9"
server_key_version: "kuduclusterkey@0"
I20260809 20:25:55.864588 24698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "5261c7f9c9e04e7fa06370f708feb55b"
format_stamp: "Formatted at 2026-08-09 20:25:55 on dist-test-slave-rnrw"
server_key: "03e6a32863fe1b13eabd4819f454eb0a"
server_key_iv: "1c2ce55b7c6ee2e053bd9b1dcc74d2e9"
server_key_version: "kuduclusterkey@0"
I20260809 20:25:55.874912 24698 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.000s
I20260809 20:25:55.915246 24720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:55.917761 24698 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.003s sys 0.002s
I20260809 20:25:55.918198 24698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "5261c7f9c9e04e7fa06370f708feb55b"
format_stamp: "Formatted at 2026-08-09 20:25:55 on dist-test-slave-rnrw"
server_key: "03e6a32863fe1b13eabd4819f454eb0a"
server_key_iv: "1c2ce55b7c6ee2e053bd9b1dcc74d2e9"
server_key_version: "kuduclusterkey@0"
I20260809 20:25:55.918641 24698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:25:55.983752 24698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:25:55.990585 24698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:25:55.991185 24698 kserver.cc:163] Server-wide thread pool size limit: 3276
Aug 09 20:25:56 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307155, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:25:56.010298 24698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:25:56.019872 24698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:25:56.020165 24698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 20:25:56.020515 24698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:25:56.020740 24698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 20:25:56 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307155, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:25:56.800989 24698 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:34283
I20260809 20:25:56.801124 24839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:34283 every 8 connection(s)
I20260809 20:25:56.803231 24698 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:25:56.814014 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 24698
I20260809 20:25:56.814291 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:25:56.815603 12600 external_mini_cluster.cc:1428] Setting key 29cc890249d43139c0976233de7ec120
Aug 09 20:25:56 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307155, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
I20260809 20:25:56.870826 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:56.813319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44883 (local address 127.12.78.1:39215)
0809 20:25:56.831185 (+ 17866us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:56.831272 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:56.841268 (+ 9996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:56.842610 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:56.842626 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:56.843582 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:25:56.844989 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:56.845024 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:56.847101 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:56.847109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:56.849937 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:56.849967 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:56.851734 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:56.851744 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:56.852154 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:56.853310 (+ 1156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:25:56.853350 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:25:56.857513 (+ 4163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:25:56.863847 (+ 6334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:25:56.863860 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:25:56.863884 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:25:56.864962 (+ 1078us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:25:56.865626 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:25:56.865632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:25:56.865636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:25:56.866057 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:25:56.867065 (+ 1008us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:25:56.867076 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:25:56.867685 (+ 609us) client_negotiation.cc:769] Sending connection context
0809 20:25:56.868079 (+ 394us) client_negotiation.cc:240] Negotiation successful
0809 20:25:56.868819 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17462,"thread_start_us":17063,"threads_started":1}
I20260809 20:25:56.872490 24843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:56.814157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:39215 (local address 127.12.78.62:44883)
0809 20:25:56.829794 (+ 15637us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:56.829801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:56.831772 (+ 1971us) server_negotiation.cc:407] Connection header received
0809 20:25:56.841713 (+ 9941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:56.841723 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:56.841834 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:56.841923 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:56.845296 (+ 3373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:56.846806 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:56.850175 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:56.851174 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:56.851310 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:56.857908 (+ 6598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:56.857918 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:56.857923 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:56.857987 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:56.863555 (+ 5568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:56.865203 (+ 1648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:56.865223 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:56.865227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:56.865436 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:56.866239 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:56.866245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:56.866248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:56.866781 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:56.866969 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:56.871562 (+ 4593us) server_negotiation.cc:299] Negotiation successful
0809 20:25:56.871883 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15455,"thread_start_us":15279,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
I20260809 20:25:56.875403 24840 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44883
I20260809 20:25:56.875883 24840 heartbeater.cc:464] Registering TS with master...
I20260809 20:25:56.877375 24840 heartbeater.cc:511] Master 127.12.78.62:44883 requested a full tablet report, sending...
I20260809 20:25:56.880820 24615 ts_manager.cc:194] Registered new tserver with Master: 5261c7f9c9e04e7fa06370f708feb55b (127.12.78.1:34283)
I20260809 20:25:56.883240 24615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:39215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:25:56.921756 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.78.20:44955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44883
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:25:57.457993 24846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:25:57.458346 24846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:25:57.458631 24846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:25:57.467185 24846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:25:57.467348 24846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:25:57.467685 24846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:25:57.479506 24846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.78.20:44955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:25:57.481081 24846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:25:57.482615 24846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:25:57.498694 24859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:57.498960 24862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:57.499029 24860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:25:57.500700 24846 server_base.cc:1034] running on GCE node
I20260809 20:25:57.501348 24846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:25:57.502810 24846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:25:57.504062 24846 hybrid_clock.cc:648] HybridClock initialized: now 1786307157504007 us; error 61 us; skew 500 ppm
I20260809 20:25:57.504434 24846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:25:57 dist-test-slave-rnrw krb5kdc[24100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307157, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:25:57.508234 24846 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:25:57.510133 24846 webserver.cc:460] Webserver started at http://127.12.78.2:41611/ using document root <none> and password file <none>
I20260809 20:25:57.511018 24846 fs_manager.cc:359] Metadata directory not provided
I20260809 20:25:57.511193 24846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:25:57.511619 24846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 20:25:57 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307157, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:25:57.530573 24846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "3ead46095deb4975a13cab740a6d3b4e"
format_stamp: "Formatted at 2026-08-09 20:25:57 on dist-test-slave-rnrw"
server_key: "d4a5ca418afecb1a8b67126323ed0381"
server_key_iv: "437ef622c5dd8d4397960211526f0ac0"
server_key_version: "kuduclusterkey@0"
I20260809 20:25:57.531812 24846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "3ead46095deb4975a13cab740a6d3b4e"
format_stamp: "Formatted at 2026-08-09 20:25:57 on dist-test-slave-rnrw"
server_key: "d4a5ca418afecb1a8b67126323ed0381"
server_key_iv: "437ef622c5dd8d4397960211526f0ac0"
server_key_version: "kuduclusterkey@0"
I20260809 20:25:57.540691 24846 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20260809 20:25:57.557487 24870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:57.559221 24846 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 20:25:57.559417 24846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "3ead46095deb4975a13cab740a6d3b4e"
format_stamp: "Formatted at 2026-08-09 20:25:57 on dist-test-slave-rnrw"
server_key: "d4a5ca418afecb1a8b67126323ed0381"
server_key_iv: "437ef622c5dd8d4397960211526f0ac0"
server_key_version: "kuduclusterkey@0"
I20260809 20:25:57.559650 24846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:25:57.585363 24846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:25:57.590461 24846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:25:57.590788 24846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:25:57.592175 24846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:25:57.594058 24846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:25:57.594193 24846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:57.594408 24846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:25:57.594561 24846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:57.658356 24846 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:33433
I20260809 20:25:57.658466 24988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:33433 every 8 connection(s)
I20260809 20:25:57.660032 24846 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:25:57.667552 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 24846
I20260809 20:25:57.667783 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:25:57.668516 12600 external_mini_cluster.cc:1428] Setting key fe8fe06ba0d4e130a14d384909c729ab
Aug 09 20:25:57 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307157, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:25:57.684206 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:57.663212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:52469 (local address 127.12.78.62:44883)
0809 20:25:57.663955 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:57.663959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:57.665707 (+ 1748us) server_negotiation.cc:407] Connection header received
0809 20:25:57.668950 (+ 3243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:57.668956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:57.669069 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:57.669212 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:57.671584 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:57.672347 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:57.674075 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:57.674477 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:57.674619 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:57.678147 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:57.678153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:57.678156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:57.678195 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:57.680974 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:57.681772 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:57.681776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:57.681779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:57.681905 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:57.682387 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:57.682391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:57.682393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:57.682699 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:57.682863 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:57.683577 (+ 714us) server_negotiation.cc:299] Negotiation successful
0809 20:25:57.683808 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":381,"threads_started":1}
I20260809 20:25:57.685278 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:57.663978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44883 (local address 127.12.78.2:52469)
0809 20:25:57.665498 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:57.665559 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:57.668517 (+ 2958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:57.669924 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:57.669937 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:57.670599 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:25:57.671401 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:57.671422 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:57.672517 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:57.672523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:57.673905 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:57.673918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:57.674728 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:57.674737 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:57.675000 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:57.675732 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:25:57.675756 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:25:57.677917 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:25:57.681134 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:25:57.681144 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:25:57.681162 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:25:57.681631 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:25:57.682022 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:25:57.682028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:25:57.682031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:25:57.682255 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:25:57.682847 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:25:57.682855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:25:57.683188 (+ 333us) client_negotiation.cc:769] Sending connection context
0809 20:25:57.683462 (+ 274us) client_negotiation.cc:240] Negotiation successful
0809 20:25:57.684023 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":807,"threads_started":1}
I20260809 20:25:57.686906 24989 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44883
I20260809 20:25:57.687256 24989 heartbeater.cc:464] Registering TS with master...
I20260809 20:25:57.688004 24989 heartbeater.cc:511] Master 127.12.78.62:44883 requested a full tablet report, sending...
I20260809 20:25:57.689869 24615 ts_manager.cc:194] Registered new tserver with Master: 3ead46095deb4975a13cab740a6d3b4e (127.12.78.2:33433)
I20260809 20:25:57.691392 24615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:52469
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:25:57.733165 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.78.20:44955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44883
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:25:57.888417 24840 heartbeater.cc:503] Master 127.12.78.62:44883 was elected leader, sending a full tablet report...
W20260809 20:25:58.196769 24996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:25:58.197094 24996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:25:58.197206 24996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:25:58.205556 24996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:25:58.205694 24996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:25:58.205818 24996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:25:58.216436 24996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.78.20:44955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:44883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:25:58.217859 24996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:25:58.219218 24996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:25:58.236043 25006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:58.235960 25004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:25:58.236359 25003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:25:58.237617 24996 server_base.cc:1034] running on GCE node
I20260809 20:25:58.238515 24996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:25:58.239777 24996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:25:58.241119 24996 hybrid_clock.cc:648] HybridClock initialized: now 1786307158241036 us; error 50 us; skew 500 ppm
I20260809 20:25:58.241576 24996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:25:58 dist-test-slave-rnrw krb5kdc[24100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307158, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:25:58.245925 24996 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:25:58.247852 24996 webserver.cc:460] Webserver started at http://127.12.78.3:37741/ using document root <none> and password file <none>
I20260809 20:25:58.248670 24996 fs_manager.cc:359] Metadata directory not provided
I20260809 20:25:58.248834 24996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:25:58.249178 24996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 20:25:58 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307158, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:25:58.264740 24996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "527356c4ee79408a8e59d8cc944e660c"
format_stamp: "Formatted at 2026-08-09 20:25:58 on dist-test-slave-rnrw"
server_key: "76c0f269eb3dc62d64adaff7cacd8e03"
server_key_iv: "39a4787282e9f3ae642cc336bf2058f7"
server_key_version: "kuduclusterkey@0"
I20260809 20:25:58.266141 24996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "527356c4ee79408a8e59d8cc944e660c"
format_stamp: "Formatted at 2026-08-09 20:25:58 on dist-test-slave-rnrw"
server_key: "76c0f269eb3dc62d64adaff7cacd8e03"
server_key_iv: "39a4787282e9f3ae642cc336bf2058f7"
server_key_version: "kuduclusterkey@0"
I20260809 20:25:58.274016 24996 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260809 20:25:58.291124 25013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:58.292943 24996 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 20:25:58.293176 24996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "527356c4ee79408a8e59d8cc944e660c"
format_stamp: "Formatted at 2026-08-09 20:25:58 on dist-test-slave-rnrw"
server_key: "76c0f269eb3dc62d64adaff7cacd8e03"
server_key_iv: "39a4787282e9f3ae642cc336bf2058f7"
server_key_version: "kuduclusterkey@0"
I20260809 20:25:58.293463 24996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:25:58.310674 24996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:25:58.315935 24996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:25:58.316478 24996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:25:58.317771 24996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:25:58.319876 24996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:25:58.320024 24996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 20:25:58.320183 24996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:25:58.320329 24996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:25:58.386098 24996 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:42169
I20260809 20:25:58.386098 25137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:42169 every 8 connection(s)
I20260809 20:25:58.387889 24996 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:25:58.394660 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 24996
I20260809 20:25:58.394944 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:25:58.395489 12600 external_mini_cluster.cc:1428] Setting key 5cead843c117ec074e8785dde0e7a429
Aug 09 20:25:58 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307158, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:25:58.415421 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.391322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:53209 (local address 127.12.78.62:44883)
0809 20:25:58.392039 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:58.392046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:58.393400 (+ 1354us) server_negotiation.cc:407] Connection header received
0809 20:25:58.396301 (+ 2901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:58.396310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:58.396411 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:58.396595 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:58.398972 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.400133 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.402444 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.403276 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.403412 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.408050 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:58.408056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:58.408059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:58.408107 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:58.411857 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:58.412875 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:58.412880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:58.412884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:58.413024 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:58.413511 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:58.413516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:58.413520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:58.413844 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:58.414061 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:58.414762 (+ 701us) server_negotiation.cc:299] Negotiation successful
0809 20:25:58.415008 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":405,"threads_started":1}
I20260809 20:25:58.416887 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.392140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:44883 (local address 127.12.78.3:53209)
0809 20:25:58.393196 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:58.393256 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:58.395873 (+ 2617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:58.396921 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:58.396937 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:58.397781 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:25:58.398738 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:58.398768 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.400324 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.400331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:58.402264 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:58.402279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.403566 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.403575 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:58.403838 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.404851 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:25:58.404876 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:25:58.407747 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:25:58.412074 (+ 4327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:25:58.412085 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:25:58.412104 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:25:58.412737 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:25:58.413144 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:25:58.413149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:25:58.413153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:25:58.413380 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:25:58.414007 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:25:58.414015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:25:58.414380 (+ 365us) client_negotiation.cc:769] Sending connection context
0809 20:25:58.414669 (+ 289us) client_negotiation.cc:240] Negotiation successful
0809 20:25:58.415240 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":502,"threads_started":1}
I20260809 20:25:58.418479 25138 heartbeater.cc:347] Connected to a master server at 127.12.78.62:44883
I20260809 20:25:58.418852 25138 heartbeater.cc:464] Registering TS with master...
I20260809 20:25:58.419983 25138 heartbeater.cc:511] Master 127.12.78.62:44883 requested a full tablet report, sending...
I20260809 20:25:58.421679 24615 ts_manager.cc:194] Registered new tserver with Master: 527356c4ee79408a8e59d8cc944e660c (127.12.78.3:42169)
I20260809 20:25:58.422519 24615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:53209
I20260809 20:25:58.425359 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:25:58.446355 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.431348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.12.78.62:44883)
0809 20:25:58.431530 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:58.431534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:58.432156 (+ 622us) server_negotiation.cc:407] Connection header received
0809 20:25:58.432401 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:58.432411 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:58.432609 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:58.432728 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:58.434345 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.435162 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.436597 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.437027 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.437165 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.440113 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:58.440123 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:58.440128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:58.440178 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:58.443251 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:58.443996 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:58.444004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:58.444007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:58.444198 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:58.444663 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:58.444672 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:58.444677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:58.445083 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:58.445342 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:58.445767 (+ 425us) server_negotiation.cc:299] Negotiation successful
0809 20:25:58.446046 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 20:25:58.451392 24615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:25:58.464617 24615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:25:58.495793 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.481249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.12.78.2:33433)
0809 20:25:58.482236 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:58.482245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:58.482984 (+ 739us) server_negotiation.cc:407] Connection header received
0809 20:25:58.483536 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:58.483546 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:58.483828 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:58.484026 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:25:58.485343 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.487361 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.490990 (+ 3629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.493105 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.493235 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.493507 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:58.495082 (+ 1575us) server_negotiation.cc:299] Negotiation successful
0809 20:25:58.495323 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":783,"threads_started":1}
I20260809 20:25:58.496837 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.481001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:33433 (local address 127.0.0.1:52426)
0809 20:25:58.482829 (+ 1828us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:58.482877 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:58.483211 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:58.484390 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:58.484401 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:58.484441 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:25:58.485076 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:58.485093 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.487618 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.487626 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:58.490770 (+ 3144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:58.490791 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.493761 (+ 2970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.493770 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:58.494132 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.494206 (+ 74us) client_negotiation.cc:769] Sending connection context
0809 20:25:58.494364 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 20:25:58.494650 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1557,"thread_start_us":1590,"threads_started":1}
I20260809 20:25:58.503989 24921 tablet_service.cc:1467] Processing CreateTablet for tablet 734bbfa7c4e74ec99daa2f9d5a436b1a (DEFAULT_TABLE table=test-table [id=977ead6fa18042edae7ae947915b6ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:25:58.505861 24921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734bbfa7c4e74ec99daa2f9d5a436b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:25:58.511299 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.481831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:42169 (local address 127.0.0.1:34020)
0809 20:25:58.497347 (+ 15516us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:58.497373 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:58.497606 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:58.501199 (+ 3593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:58.501204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:58.501232 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:25:58.501526 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:58.501542 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.504529 (+ 2987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.504534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:58.506842 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:58.506856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.510326 (+ 3470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.510339 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:58.510541 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.510578 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 20:25:58.510654 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 20:25:58.510878 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15346,"thread_start_us":21465,"threads_started":1}
I20260809 20:25:58.513172 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.484314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.12.78.1:34283)
0809 20:25:58.493166 (+ 8852us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:58.493180 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:58.498428 (+ 5248us) server_negotiation.cc:407] Connection header received
0809 20:25:58.498633 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:58.498647 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:58.499067 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:58.499324 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:25:58.502446 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.504156 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.508578 (+ 4422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.510856 (+ 2278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.510999 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.511298 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:58.512439 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0809 20:25:58.512688 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8543,"thread_start_us":8369,"threads_started":1}
I20260809 20:25:58.513598 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.482313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.12.78.3:42169)
0809 20:25:58.499260 (+ 16947us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:58.499268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:58.499330 (+ 62us) server_negotiation.cc:407] Connection header received
0809 20:25:58.500008 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:58.500016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:58.500301 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:58.500773 (+ 472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:25:58.502591 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.504315 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.508457 (+ 4142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.510087 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.512422 (+ 2335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.512715 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:58.512790 (+ 75us) server_negotiation.cc:299] Negotiation successful
0809 20:25:58.512979 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16704,"thread_start_us":17051,"threads_started":1}
I20260809 20:25:58.514078 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.483606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:34283 (local address 127.0.0.1:32898)
0809 20:25:58.498275 (+ 14669us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:58.498292 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:58.498444 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:58.499689 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:58.499697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:58.499742 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:25:58.502204 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:58.502221 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.504315 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.504322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:58.508260 (+ 3938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:58.508275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.511938 (+ 3663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.511946 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:58.512196 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.512230 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 20:25:58.513398 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0809 20:25:58.513553 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14545,"mutex_wait_us":87,"thread_start_us":14371,"threads_started":1}
I20260809 20:25:58.518313 24771 tablet_service.cc:1467] Processing CreateTablet for tablet 734bbfa7c4e74ec99daa2f9d5a436b1a (DEFAULT_TABLE table=test-table [id=977ead6fa18042edae7ae947915b6ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:25:58.518638 25069 tablet_service.cc:1467] Processing CreateTablet for tablet 734bbfa7c4e74ec99daa2f9d5a436b1a (DEFAULT_TABLE table=test-table [id=977ead6fa18042edae7ae947915b6ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:25:58.520236 24771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734bbfa7c4e74ec99daa2f9d5a436b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:25:58.520443 25069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734bbfa7c4e74ec99daa2f9d5a436b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:25:58.537495 25159 tablet_bootstrap.cc:492] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e: Bootstrap starting.
I20260809 20:25:58.541980 25159 tablet_bootstrap.cc:654] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e: Neither blocks nor log segments found. Creating new log.
I20260809 20:25:58.543516 25159 log.cc:824] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e: Log is configured to *not* fsync() on all Append() calls
I20260809 20:25:58.547340 25160 tablet_bootstrap.cc:492] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b: Bootstrap starting.
I20260809 20:25:58.550575 25159 tablet_bootstrap.cc:492] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e: No bootstrap required, opened a new log
I20260809 20:25:58.551028 25159 ts_tablet_manager.cc:1397] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260809 20:25:58.554714 25160 tablet_bootstrap.cc:654] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b: Neither blocks nor log segments found. Creating new log.
I20260809 20:25:58.556785 25160 log.cc:824] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b: Log is configured to *not* fsync() on all Append() calls
I20260809 20:25:58.559301 25161 tablet_bootstrap.cc:492] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c: Bootstrap starting.
I20260809 20:25:58.564227 25159 raft_consensus.cc:348] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ead46095deb4975a13cab740a6d3b4e" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 33433 } } peers { permanent_uuid: "5261c7f9c9e04e7fa06370f708feb55b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34283 } } peers { permanent_uuid: "527356c4ee79408a8e59d8cc944e660c" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 42169 } }
I20260809 20:25:58.565042 25161 tablet_bootstrap.cc:654] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c: Neither blocks nor log segments found. Creating new log.
I20260809 20:25:58.565708 25159 raft_consensus.cc:374] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:25:58.565973 25159 raft_consensus.cc:729] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ead46095deb4975a13cab740a6d3b4e, State: Initialized, Role: FOLLOWER
I20260809 20:25:58.567025 25161 log.cc:824] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c: Log is configured to *not* fsync() on all Append() calls
I20260809 20:25:58.566879 25159 consensus_queue.cc:260] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ead46095deb4975a13cab740a6d3b4e" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 33433 } } peers { permanent_uuid: "5261c7f9c9e04e7fa06370f708feb55b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34283 } } peers { permanent_uuid: "527356c4ee79408a8e59d8cc944e660c" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 42169 } }
I20260809 20:25:58.569222 25160 tablet_bootstrap.cc:492] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b: No bootstrap required, opened a new log
I20260809 20:25:58.569597 25160 ts_tablet_manager.cc:1397] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.007s
I20260809 20:25:58.581307 25160 raft_consensus.cc:348] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ead46095deb4975a13cab740a6d3b4e" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 33433 } } peers { permanent_uuid: "5261c7f9c9e04e7fa06370f708feb55b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34283 } } peers { permanent_uuid: "527356c4ee79408a8e59d8cc944e660c" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 42169 } }
I20260809 20:25:58.583220 25160 raft_consensus.cc:374] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:25:58.583417 25160 raft_consensus.cc:729] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5261c7f9c9e04e7fa06370f708feb55b, State: Initialized, Role: FOLLOWER
I20260809 20:25:58.583678 24989 heartbeater.cc:503] Master 127.12.78.62:44883 was elected leader, sending a full tablet report...
I20260809 20:25:58.584141 25159 ts_tablet_manager.cc:1428] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e: Time spent starting tablet: real 0.033s user 0.010s sys 0.021s
I20260809 20:25:58.584259 25160 consensus_queue.cc:260] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ead46095deb4975a13cab740a6d3b4e" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 33433 } } peers { permanent_uuid: "5261c7f9c9e04e7fa06370f708feb55b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34283 } } peers { permanent_uuid: "527356c4ee79408a8e59d8cc944e660c" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 42169 } }
I20260809 20:25:58.587983 25161 tablet_bootstrap.cc:492] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c: No bootstrap required, opened a new log
I20260809 20:25:58.588480 25161 ts_tablet_manager.cc:1397] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.014s
I20260809 20:25:58.591935 25160 ts_tablet_manager.cc:1428] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b: Time spent starting tablet: real 0.022s user 0.008s sys 0.011s
I20260809 20:25:58.602859 25161 raft_consensus.cc:348] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ead46095deb4975a13cab740a6d3b4e" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 33433 } } peers { permanent_uuid: "5261c7f9c9e04e7fa06370f708feb55b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34283 } } peers { permanent_uuid: "527356c4ee79408a8e59d8cc944e660c" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 42169 } }
I20260809 20:25:58.603269 25161 raft_consensus.cc:374] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:25:58.603407 25161 raft_consensus.cc:729] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527356c4ee79408a8e59d8cc944e660c, State: Initialized, Role: FOLLOWER
I20260809 20:25:58.604122 25161 consensus_queue.cc:260] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ead46095deb4975a13cab740a6d3b4e" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 33433 } } peers { permanent_uuid: "5261c7f9c9e04e7fa06370f708feb55b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34283 } } peers { permanent_uuid: "527356c4ee79408a8e59d8cc944e660c" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 42169 } }
I20260809 20:25:58.605450 25138 heartbeater.cc:503] Master 127.12.78.62:44883 was elected leader, sending a full tablet report...
I20260809 20:25:58.606127 25161 ts_tablet_manager.cc:1428] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c: Time spent starting tablet: real 0.017s user 0.011s sys 0.001s
W20260809 20:25:58.640618 25139 tablet.cc:2369] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:25:58.663130 24990 tablet.cc:2369] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:25:58.822767 24841 tablet.cc:2369] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:25:58.827088 25167 raft_consensus.cc:482] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:25:58.827464 25167 raft_consensus.cc:504] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ead46095deb4975a13cab740a6d3b4e" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 33433 } } peers { permanent_uuid: "5261c7f9c9e04e7fa06370f708feb55b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34283 } } peers { permanent_uuid: "527356c4ee79408a8e59d8cc944e660c" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 42169 } }
I20260809 20:25:58.829600 25167 leader_election.cc:288] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ead46095deb4975a13cab740a6d3b4e (127.12.78.2:33433), 5261c7f9c9e04e7fa06370f708feb55b (127.12.78.1:34283)
I20260809 20:25:58.837010 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.830098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:45549 (local address 127.12.78.2:33433)
0809 20:25:58.830305 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:58.830310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:58.830357 (+ 47us) server_negotiation.cc:407] Connection header received
0809 20:25:58.830426 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:58.830430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:58.830528 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:58.830638 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:25:58.831292 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.832437 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.834822 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.835974 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.836080 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.836160 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:58.836657 (+ 497us) server_negotiation.cc:299] Negotiation successful
0809 20:25:58.836759 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 20:25:58.837194 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.829908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:33433 (local address 127.12.78.3:45549)
0809 20:25:58.830085 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:58.830107 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:58.830252 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:58.830840 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:58.830847 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:58.830887 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:25:58.831141 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:58.831154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.832648 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.832655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:58.834641 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:58.834655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.836147 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.836154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:58.836399 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.836439 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 20:25:58.836639 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 20:25:58.836873 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 20:25:58.838201 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.830738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:58925 (local address 127.12.78.1:34283)
0809 20:25:58.830977 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:58.830983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:58.831003 (+ 20us) server_negotiation.cc:407] Connection header received
0809 20:25:58.831170 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:58.831174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:58.831258 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:58.831348 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:25:58.832156 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.833386 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.835740 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.837236 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.837415 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.837532 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:58.837884 (+ 352us) server_negotiation.cc:299] Negotiation successful
0809 20:25:58.838015 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 20:25:58.838340 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.830324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:34283 (local address 127.12.78.3:58925)
0809 20:25:58.830925 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0809 20:25:58.830940 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 20:25:58.831035 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:25:58.831406 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:25:58.831414 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:25:58.831440 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:25:58.831959 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:58.831972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.833547 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.833551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:58.835556 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:25:58.835571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.837508 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.837514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:25:58.837719 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.837743 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 20:25:58.837817 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 20:25:58.837941 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":1,"thread_start_us":328,"threads_started":1}
I20260809 20:25:58.838577 24942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734bbfa7c4e74ec99daa2f9d5a436b1a" candidate_uuid: "527356c4ee79408a8e59d8cc944e660c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ead46095deb4975a13cab740a6d3b4e" is_pre_election: true
I20260809 20:25:58.839536 24942 raft_consensus.cc:2454] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 527356c4ee79408a8e59d8cc944e660c in term 0.
I20260809 20:25:58.839733 24793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734bbfa7c4e74ec99daa2f9d5a436b1a" candidate_uuid: "527356c4ee79408a8e59d8cc944e660c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5261c7f9c9e04e7fa06370f708feb55b" is_pre_election: true
I20260809 20:25:58.840364 24793 raft_consensus.cc:2454] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 527356c4ee79408a8e59d8cc944e660c in term 0.
I20260809 20:25:58.840498 25015 leader_election.cc:302] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ead46095deb4975a13cab740a6d3b4e, 527356c4ee79408a8e59d8cc944e660c; no voters:
I20260809 20:25:58.841147 25167 raft_consensus.cc:2781] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:25:58.841354 25167 raft_consensus.cc:482] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:25:58.841487 25167 raft_consensus.cc:3037] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:25:58.844116 25167 raft_consensus.cc:504] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ead46095deb4975a13cab740a6d3b4e" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 33433 } } peers { permanent_uuid: "5261c7f9c9e04e7fa06370f708feb55b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34283 } } peers { permanent_uuid: "527356c4ee79408a8e59d8cc944e660c" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 42169 } }
I20260809 20:25:58.844892 25167 leader_election.cc:288] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [CANDIDATE]: Term 1 election: Requested vote from peers 3ead46095deb4975a13cab740a6d3b4e (127.12.78.2:33433), 5261c7f9c9e04e7fa06370f708feb55b (127.12.78.1:34283)
I20260809 20:25:58.845479 24942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734bbfa7c4e74ec99daa2f9d5a436b1a" candidate_uuid: "527356c4ee79408a8e59d8cc944e660c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ead46095deb4975a13cab740a6d3b4e"
I20260809 20:25:58.845523 24793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734bbfa7c4e74ec99daa2f9d5a436b1a" candidate_uuid: "527356c4ee79408a8e59d8cc944e660c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5261c7f9c9e04e7fa06370f708feb55b"
I20260809 20:25:58.845767 24942 raft_consensus.cc:3037] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:25:58.845937 24793 raft_consensus.cc:3037] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:25:58.848398 24942 raft_consensus.cc:2454] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527356c4ee79408a8e59d8cc944e660c in term 1.
I20260809 20:25:58.848842 24793 raft_consensus.cc:2454] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527356c4ee79408a8e59d8cc944e660c in term 1.
I20260809 20:25:58.848986 25015 leader_election.cc:302] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ead46095deb4975a13cab740a6d3b4e, 527356c4ee79408a8e59d8cc944e660c; no voters:
I20260809 20:25:58.849362 25167 raft_consensus.cc:2781] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:25:58.850003 25167 raft_consensus.cc:686] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [term 1 LEADER]: Becoming Leader. State: Replica: 527356c4ee79408a8e59d8cc944e660c, State: Running, Role: LEADER
I20260809 20:25:58.850618 25167 consensus_queue.cc:237] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ead46095deb4975a13cab740a6d3b4e" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 33433 } } peers { permanent_uuid: "5261c7f9c9e04e7fa06370f708feb55b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34283 } } peers { permanent_uuid: "527356c4ee79408a8e59d8cc944e660c" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 42169 } }
I20260809 20:25:58.855593 24615 catalog_manager.cc:5495] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c reported cstate change: term changed from 0 to 1, leader changed from <none> to 527356c4ee79408a8e59d8cc944e660c (127.12.78.3). New cstate: current_term: 1 leader_uuid: "527356c4ee79408a8e59d8cc944e660c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ead46095deb4975a13cab740a6d3b4e" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 33433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5261c7f9c9e04e7fa06370f708feb55b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "527356c4ee79408a8e59d8cc944e660c" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 42169 } health_report { overall_health: HEALTHY } } }
I20260809 20:25:58.890992 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.885318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.12.78.3:42169)
0809 20:25:58.885525 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:58.885531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:58.885549 (+ 18us) server_negotiation.cc:407] Connection header received
0809 20:25:58.885673 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:58.885676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:58.885762 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:58.885891 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:25:58.886276 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.887003 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.888715 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.889154 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.889266 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:58.889820 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:25:58.890369 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:25:58.890488 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:58.890606 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 20:25:58.890681 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 20:25:58.900610 24942 raft_consensus.cc:1261] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e [term 1 FOLLOWER]: Refusing update from remote peer 527356c4ee79408a8e59d8cc944e660c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:25:58.900724 24793 raft_consensus.cc:1261] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b [term 1 FOLLOWER]: Refusing update from remote peer 527356c4ee79408a8e59d8cc944e660c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:25:58.901855 25169 consensus_queue.cc:1035] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5261c7f9c9e04e7fa06370f708feb55b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:25:58.902295 25167 consensus_queue.cc:1035] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ead46095deb4975a13cab740a6d3b4e" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 33433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:25:58.916693 25172 mvcc.cc:205] Tried to move back new op lower bound from 7316714122845249536 to 7316714122656833536. Current Snapshot: MvccSnapshot[applied={T|T < 7316714122845249536}]
I20260809 20:25:58.935715 24614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:25:58.936123 24614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:25:58.940647 24614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b71ba63417e463296dfad38b5b6da40: Sending DeleteTablet for 3 replicas of tablet 734bbfa7c4e74ec99daa2f9d5a436b1a
I20260809 20:25:58.942137 24771 tablet_service.cc:1514] Processing DeleteTablet for tablet 734bbfa7c4e74ec99daa2f9d5a436b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:25:58 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:32898
I20260809 20:25:58.942080 24921 tablet_service.cc:1514] Processing DeleteTablet for tablet 734bbfa7c4e74ec99daa2f9d5a436b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:25:58 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:52426
I20260809 20:25:58.942401 25069 tablet_service.cc:1514] Processing DeleteTablet for tablet 734bbfa7c4e74ec99daa2f9d5a436b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:25:58 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:34020
I20260809 20:25:58.945536 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 24698
I20260809 20:25:58.975241 25184 tablet_replica.cc:330] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b: stopping tablet replica
I20260809 20:25:58.976148 25184 raft_consensus.cc:2229] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:25:58.982434 25184 raft_consensus.cc:2258] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:25:58.985903 25184 ts_tablet_manager.cc:1905] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:25:58.989416 25183 tablet_replica.cc:330] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e: stopping tablet replica
I20260809 20:25:58.989861 25185 tablet_replica.cc:330] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c: stopping tablet replica
I20260809 20:25:58.990080 25183 raft_consensus.cc:2229] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:25:58.990486 25185 raft_consensus.cc:2229] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:25:58.990907 25183 raft_consensus.cc:2258] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:25:58.991418 25185 raft_consensus.cc:2258] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:25:58.993801 25183 ts_tablet_manager.cc:1905] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 20:25:58 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:25:58.998369 25185 ts_tablet_manager.cc:1905] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:25:58.998528 25184 ts_tablet_manager.cc:1918] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:25:58.999080 25184 log.cc:1196] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/734bbfa7c4e74ec99daa2f9d5a436b1a
I20260809 20:25:58.999886 25184 ts_tablet_manager.cc:1939] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 5261c7f9c9e04e7fa06370f708feb55b: Deleting consensus metadata
I20260809 20:25:59.002996 24599 catalog_manager.cc:4841] TS 5261c7f9c9e04e7fa06370f708feb55b (127.12.78.1:34283): tablet 734bbfa7c4e74ec99daa2f9d5a436b1a (table test-table [id=977ead6fa18042edae7ae947915b6ec7]) successfully deleted
I20260809 20:25:59.004923 25183 ts_tablet_manager.cc:1918] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:25:59.005182 25183 log.cc:1196] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/734bbfa7c4e74ec99daa2f9d5a436b1a
I20260809 20:25:59.005882 25183 ts_tablet_manager.cc:1939] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 3ead46095deb4975a13cab740a6d3b4e: Deleting consensus metadata
I20260809 20:25:59.008303 24599 catalog_manager.cc:4841] TS 3ead46095deb4975a13cab740a6d3b4e (127.12.78.2:33433): tablet 734bbfa7c4e74ec99daa2f9d5a436b1a (table test-table [id=977ead6fa18042edae7ae947915b6ec7]) successfully deleted
I20260809 20:25:59.009805 25185 ts_tablet_manager.cc:1918] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:25:59.010044 25185 log.cc:1196] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/734bbfa7c4e74ec99daa2f9d5a436b1a
I20260809 20:25:59.010936 25185 ts_tablet_manager.cc:1939] T 734bbfa7c4e74ec99daa2f9d5a436b1a P 527356c4ee79408a8e59d8cc944e660c: Deleting consensus metadata
I20260809 20:25:59.011361 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:58.947100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.12.78.1:34283)
0809 20:25:58.947340 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:58.947349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:58.971133 (+ 23784us) server_negotiation.cc:407] Connection header received
0809 20:25:58.971775 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:58.971782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:58.971920 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:58.972027 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:58.976609 (+ 4582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.978515 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.986309 (+ 7794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:58.987278 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:58.987424 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:59.000261 (+ 12837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:59.000279 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:59.000299 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:59.000351 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:59.005719 (+ 5368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:59.006973 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:59.006984 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:59.006989 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:59.007186 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:59.007811 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:59.007817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:59.007820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:59.009026 (+ 1206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:59.009408 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:59.010311 (+ 903us) server_negotiation.cc:299] Negotiation successful
0809 20:25:59.010802 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 20:25:59.012149 24835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:25:59.013276 24601 catalog_manager.cc:4841] TS 527356c4ee79408a8e59d8cc944e660c (127.12.78.3:42169): tablet 734bbfa7c4e74ec99daa2f9d5a436b1a (table test-table [id=977ead6fa18042edae7ae947915b6ec7]) successfully deleted
I20260809 20:25:59.119932 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 24698
I20260809 20:25:59.147635 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 24846
Aug 09 20:25:59 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:25:59.166461 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:59.148534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52428 (local address 127.12.78.2:33433)
0809 20:25:59.148729 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:59.148733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:59.148753 (+ 20us) server_negotiation.cc:407] Connection header received
0809 20:25:59.148957 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:59.148961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:59.149035 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:59.149135 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:59.150943 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:59.151715 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:59.153833 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:59.154256 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:59.154358 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:59.159876 (+ 5518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:59.159893 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:59.159909 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:59.159946 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:59.163049 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:59.163943 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:59.163951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:59.163953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:59.164115 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:59.164595 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:59.164601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:59.164605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:59.165112 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:59.165430 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:59.165796 (+ 366us) server_negotiation.cc:299] Negotiation successful
0809 20:25:59.166071 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 20:25:59.172712 24984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:25:59.277067 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 24846
I20260809 20:25:59.300114 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 24996
Aug 09 20:25:59 dist-test-slave-rnrw krb5kdc[24100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:25:59.318744 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:25:59.301038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.12.78.3:42169)
0809 20:25:59.301521 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0809 20:25:59.301525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:25:59.301542 (+ 17us) server_negotiation.cc:407] Connection header received
0809 20:25:59.301597 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:25:59.301600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:25:59.301669 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:25:59.301830 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:25:59.303324 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:59.304208 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:59.305662 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:25:59.306145 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:25:59.306296 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:25:59.311531 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:25:59.311546 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:25:59.311565 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:25:59.311605 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:25:59.315051 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:59.316098 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:59.316107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:59.316110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:59.316301 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:25:59.316872 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:25:59.316876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:25:59.316879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:25:59.317292 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:25:59.317637 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0809 20:25:59.318091 (+ 454us) server_negotiation.cc:299] Negotiation successful
0809 20:25:59.318365 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368}
I20260809 20:25:59.319591 25132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:25:59.448746 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 24996
I20260809 20:25:59.473214 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 24580
I20260809 20:25:59.474648 24647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:25:59.589664 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 24580
I20260809 20:25:59.614876 12600 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 20:25:59.950700 12600 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 20:25:59.950975 12600 mini_ranger.cc:66] Stopping Ranger...
I20260809 20:26:00.297108 12600 mini_ranger.cc:68] Stopped Ranger
2026-08-09 20:26:00.297 UTC [24131] LOG: received smart shutdown request
2026-08-09 20:26:00.301 UTC [24131] LOG: background worker "logical replication launcher" (PID 24140) exited with exit code 1
2026-08-09 20:26:00.301 UTC [24135] LOG: shutting down
2026-08-09 20:26:00.481 UTC [24131] LOG: database system is shut down
2026-08-09T20:26:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (55352 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:26:00 dist-test-slave-rnrw krb5kdc[25201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:26:00 dist-test-slave-rnrw krb5kdc[25201](info): set up 2 sockets
Aug 09 20:26:00 dist-test-slave-rnrw krb5kdc[25201](info): commencing operation
krb5kdc: starting...
W20260809 20:26:01.785851 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.006s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:26:01 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:26:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:26:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.78.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.78.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/rangeradmin_127.12.78.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/rangeradmin_127.12.78.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.78.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.78.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/rangerlookup_127.12.78.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/rangerlookup_127.12.78.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.78.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.78.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/HTTP_127.12.78.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/HTTP_127.12.78.20@KRBTEST.COM.keytab.
I20260809 20:26:02.184473 12600 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktNpCrv/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/postgres -l logfile start
2026-08-09 20:26:04.813 UTC [25232] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 20:26:04.813 UTC [25232] LOG: listening on IPv4 address "127.12.78.20", port 46099
2026-08-09 20:26:04.816 UTC [25232] LOG: listening on Unix socket "/tmp/.s.PGSQL.46099"
2026-08-09 20:26:04.841 UTC [25235] LOG: database system was shut down at 2026-08-09 20:26:03 UTC
2026-08-09 20:26:04.844 UTC [25232] LOG: database system is ready to accept connections
I20260809 20:26:05.892427 12600 mini_postgres.cc:96] Postgres bound to 46099
2026-08-09 20:26:05.914 UTC [25244] FATAL: database "slave" does not exist
127.12.78.20:46099 - accepting connections
I20260809 20:26:05.915457 12600 mini_ranger.cc:161] Starting Ranger...
I20260809 20:26:05.946861 12600 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 20:26:06.451757 12600 mini_ranger.cc:87] Created ranger Postgres database
I20260809 20:26:06.451890 12600 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-admin
2026-08-09 20:26:06,764 [I] DB FLAVOR :POSTGRES
2026-08-09 20:26:06,765 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 20:26:06,765 [I] Checking connection..
2026-08-09 20:26:06,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 20:26:07,044 [I] Checking connection passed.
2026-08-09 20:26:07,045 [I] --------- Verifying version history table ---------
2026-08-09 20:26:07,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 20:26:07,321 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 20:26:07,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 20:26:07,579 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 20:26:07,579 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 20:26:07,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 20:26:07.840 UTC [25310] WARNING: there is no transaction in progress
2026-08-09 20:26:07,848 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 20:26:07,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 20:26:08,100 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 20:26:08,100 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 20:26:08,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 20:26:08,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 20:26:08,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rnrw.c.gcp-upstream.internal','N') ;"
2026-08-09 20:26:08,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 20:26:09,159 [I] Table x_portal_user does not exist in database ranger
2026-08-09 20:26:09,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 20:26:09,415 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 20:26:09,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 20:26:09,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 20:26:09,928 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 20:26:09,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 20:26:10.968 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:10.985 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.001 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.018 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.033 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.051 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.110 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.116 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.131 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.142 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.153 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.168 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.178 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.185 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.192 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11.199 UTC [25427] WARNING: there is no transaction in progress
2026-08-09 20:26:11,891 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 20:26:11,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 20:26:12,142 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 20:26:12,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 20:26:12,403 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 20:26:12,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 20:26:12,651 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 20:26:12,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 20:26:12,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 20:26:13,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rnrw.c.gcp-upstream.internal';"
2026-08-09 20:26:13,414 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 20:26:13,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 20:26:13,686 [I] DB_PATCHES have already been applied
I20260809 20:26:13.692102 12600 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktNpCrv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 20:26:13.692200 12600 mini_ranger.cc:193] Using host: 127.12.78.20
I20260809 20:26:13.711099 12600 mini_ranger.cc:227] Ranger admin URL: http://127.12.78.20:37345
Aug 09, 2026 8:26:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 8:26:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 8:26:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 8:26:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 8:26:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 8:26:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 8:26:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 20:26:14.952281 12600 mini_ranger.cc:160] Time spent starting Ranger: real 9.037s user 0.004s sys 0.049s
Aug 09, 2026 8:26:36 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 8:26:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 09, 2026 8:26:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 8:26:36 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 8:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 8:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 8:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 8:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 8:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 8:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 8:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 8:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 8:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 8:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 8:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 8:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 8:26:36 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 8:26:37 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 20:26:37.634212 12600 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.78.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.78.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/rangerkms_127.12.78.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.78.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/rangerkms_127.12.78.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.78.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.78.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.78.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/HTTP_127.12.78.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.78.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/HTTP_127.12.78.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 20:26:37 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307197, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 20:26:37.831179 12600 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 20:26:37.857326 12600 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 20:26:38.739434 12600 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 20:26:38.739542 12600 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-kms
I20260809 20:26:38.739580 12600 mini_ranger_kms.cc:227] Using postgres at 127.12.78.20:46099
2026-08-09 20:26:39,118 [I] DB FLAVOR :POSTGRES
2026-08-09 20:26:39,119 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 20:26:39,119 [I] Checking connection
2026-08-09 20:26:39,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 20:26:39,377 [I] connection success
2026-08-09 20:26:39,377 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 20:26:39,377 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 20:26:39,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 20:26:39,627 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 20:26:39,627 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 20:26:39,628 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 20:26:39,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.78.20:46099/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 20:26:39,900 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 20:26:40.414315 12600 mini_ranger_kms.cc:310] Created kms service
I20260809 20:26:40.685194 12600 mini_ranger_kms.cc:326] Created kudu user
I20260809 20:26:40.734999 12600 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 20:26:40 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307197, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:26:41.199901 12600 mini_ranger_kms.cc:384] Added ranger policy
I20260809 20:26:41.200224 12600 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktNpCrv/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktNpCrv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 20:26:41.200287 12600 mini_ranger_kms.cc:240] Using host: 127.12.78.20
I20260809 20:26:41.224349 12600 mini_ranger_kms.cc:276] Ranger KMS PID: 25617
I20260809 20:26:41.224510 12600 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.78.20:57479
Aug 09, 2026 8:26:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 8:26:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 8:26:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 8:26:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 8:26:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 8:26:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 20:26:43 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786307203, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.78.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:26:43.651109 12600 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.820s user 0.057s sys 0.354s
I20260809 20:26:43.651419 12600 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 20:26:43.651654 12600 mini_ranger_kms.cc:401] 127.12.78.20:57479/kms/v1/keys
Aug 09 20:26:43 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307203, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.78.20@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
Aug 09 20:26:44 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307203, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.78.20@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
Aug 09 20:26:46 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:26:46.688936 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40977
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:40977
--ranger_config_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.78.20:57479/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:26:47.063541 25673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:26:47.063853 25673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:26:47.064013 25673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:26:47.071391 25673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:26:47.071516 25673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:26:47.071642 25673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:26:47.071754 25673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:26:47.071825 25673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:26:47.082922 25673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.78.20:57479/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:40977
--ranger_config_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:26:47.084316 25673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:26:47.085739 25673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:26:47.098743 25678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:47.098706 25679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:47.098851 25681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:47.099604 25673 server_base.cc:1034] running on GCE node
I20260809 20:26:47.100454 25673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:26:47.102020 25673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:26:47.103281 25673 hybrid_clock.cc:648] HybridClock initialized: now 1786307207103238 us; error 46 us; skew 500 ppm
I20260809 20:26:47.103670 25673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:26:47 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:26:47.107463 25673 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:26:47.109241 25673 webserver.cc:460] Webserver started at http://127.12.78.62:46403/ using document root <none> and password file <none>
I20260809 20:26:47.109941 25673 fs_manager.cc:359] Metadata directory not provided
I20260809 20:26:47.110093 25673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:26:47.110440 25673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 20:26:47 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:26:47.180521 25673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "a0653716cdf84d8c9f767c29eae93792"
format_stamp: "Formatted at 2026-08-09 20:26:47 on dist-test-slave-rnrw"
server_key: "82be749a0894083f5cc481b5ba21c556"
server_key_iv: "e65f154eb80240dd39eac147635fc480"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:47.181572 25673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "a0653716cdf84d8c9f767c29eae93792"
format_stamp: "Formatted at 2026-08-09 20:26:47 on dist-test-slave-rnrw"
server_key: "82be749a0894083f5cc481b5ba21c556"
server_key_iv: "e65f154eb80240dd39eac147635fc480"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:47.187813 25673 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260809 20:26:47.209920 25689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:47.211740 25673 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 20:26:47.211966 25673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "a0653716cdf84d8c9f767c29eae93792"
format_stamp: "Formatted at 2026-08-09 20:26:47 on dist-test-slave-rnrw"
server_key: "82be749a0894083f5cc481b5ba21c556"
server_key_iv: "e65f154eb80240dd39eac147635fc480"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:47.212230 25673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:26:47.230495 25673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:26:47.235265 25673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:26:47.235653 25673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:26:47.257710 25673 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:40977
I20260809 20:26:47.257726 25742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:40977 every 8 connection(s)
I20260809 20:26:47.259366 25673 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:26:47.262226 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 25673
I20260809 20:26:47.262462 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:26:47.263319 12600 external_mini_cluster.cc:1428] Setting key a8945eb022be221576eeab9f900bef7c
I20260809 20:26:47.264132 25743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:26:47.284240 25743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792: Bootstrap starting.
I20260809 20:26:47.288458 25743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792: Neither blocks nor log segments found. Creating new log.
I20260809 20:26:47.289601 25743 log.cc:824] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792: Log is configured to *not* fsync() on all Append() calls
I20260809 20:26:47.293316 25743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792: No bootstrap required, opened a new log
Aug 09 20:26:47 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:26:47.303624 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:47.265590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.12.78.62:40977)
0809 20:26:47.277224 (+ 11634us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:47.277239 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:47.279758 (+ 2519us) server_negotiation.cc:407] Connection header received
0809 20:26:47.282686 (+ 2928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:47.282721 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:47.283323 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:47.283833 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:47.285577 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:47.287689 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:47.289714 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:47.290770 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:47.291019 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:47.295492 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:47.295501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:47.295517 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:47.295563 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:47.298877 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:47.299739 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:47.299746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:47.299754 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:47.299938 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:47.300383 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:47.300390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:47.300401 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:47.301122 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:47.301487 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:47.301993 (+ 506us) server_negotiation.cc:299] Negotiation successful
0809 20:26:47.302508 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11359,"thread_start_us":11094,"threads_started":1}
I20260809 20:26:47.304643 25743 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0653716cdf84d8c9f767c29eae93792" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40977 } }
I20260809 20:26:47.305204 25743 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:26:47.305379 25743 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0653716cdf84d8c9f767c29eae93792, State: Initialized, Role: FOLLOWER
I20260809 20:26:47.306031 25743 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0653716cdf84d8c9f767c29eae93792" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40977 } }
I20260809 20:26:47.306316 25743 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:26:47.306469 25743 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:26:47.306658 25743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:26:47.310114 25743 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0653716cdf84d8c9f767c29eae93792" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40977 } }
I20260809 20:26:47.310721 25743 leader_election.cc:302] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0653716cdf84d8c9f767c29eae93792; no voters:
I20260809 20:26:47.311497 25743 leader_election.cc:288] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:26:47.311642 25748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:26:47.313802 25743 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:26:47.313808 25748 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 1 LEADER]: Becoming Leader. State: Replica: a0653716cdf84d8c9f767c29eae93792, State: Running, Role: LEADER
I20260809 20:26:47.314429 25748 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0653716cdf84d8c9f767c29eae93792" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40977 } }
I20260809 20:26:47.317479 25749 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0653716cdf84d8c9f767c29eae93792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0653716cdf84d8c9f767c29eae93792" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40977 } } }
I20260809 20:26:47.317915 25749 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [sys.catalog]: This master's current role is: LEADER
I20260809 20:26:47.317515 25750 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0653716cdf84d8c9f767c29eae93792. Latest consensus state: current_term: 1 leader_uuid: "a0653716cdf84d8c9f767c29eae93792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0653716cdf84d8c9f767c29eae93792" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40977 } } }
I20260809 20:26:47.319314 25750 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [sys.catalog]: This master's current role is: LEADER
I20260809 20:26:47.337438 25754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:26:47.345856 25754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:26:47.371685 25754 catalog_manager.cc:1347] Generated new cluster ID: a614185df1af45ca93a19e7700d5c5f2
I20260809 20:26:47.371833 25754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:26:47.391197 25754 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:26:47.393154 25754 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:26:47.400601 25743 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 20:26:47.402225 25754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792: Generated new TSK 0
I20260809 20:26:47.403218 25754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:26:48.283751 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.78.20:57479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40977
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 20:26:48 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786307208, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 20:26:48.737536 25791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:26:48.737862 25791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:26:48.738032 25791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:26:48.746158 25791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:26:48.746320 25791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:26:48.746547 25791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:26:48.759553 25791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.78.20:57479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:26:48.761555 25791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:26:48.763190 25791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:26:48.799880 25802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:48.802155 25803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:48.813658 25791 server_base.cc:1034] running on GCE node
W20260809 20:26:48.815047 25805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:48.816707 25791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:26:48.818993 25791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:26:48.820340 25791 hybrid_clock.cc:648] HybridClock initialized: now 1786307208820254 us; error 69 us; skew 500 ppm
I20260809 20:26:48.820844 25791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:26:48 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307208, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:26:48.826469 25791 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:26:48.832522 25791 webserver.cc:460] Webserver started at http://127.12.78.1:33039/ using document root <none> and password file <none>
I20260809 20:26:48.834100 25791 fs_manager.cc:359] Metadata directory not provided
I20260809 20:26:48.834306 25791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:26:48.834648 25791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 20:26:48 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307208, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:26:48.859237 25791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "d42c4ce4de254a5382d23c5e01f7a513"
format_stamp: "Formatted at 2026-08-09 20:26:48 on dist-test-slave-rnrw"
server_key: "21740a3b422e31e108072712dbb98139"
server_key_iv: "9e5c36079ce4685080a409e16ffc1406"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:48.860687 25791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "d42c4ce4de254a5382d23c5e01f7a513"
format_stamp: "Formatted at 2026-08-09 20:26:48 on dist-test-slave-rnrw"
server_key: "21740a3b422e31e108072712dbb98139"
server_key_iv: "9e5c36079ce4685080a409e16ffc1406"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:48.877892 25791 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.008s sys 0.004s
I20260809 20:26:48.899344 25812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:48.901510 25791 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 20:26:48.901732 25791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "d42c4ce4de254a5382d23c5e01f7a513"
format_stamp: "Formatted at 2026-08-09 20:26:48 on dist-test-slave-rnrw"
server_key: "21740a3b422e31e108072712dbb98139"
server_key_iv: "9e5c36079ce4685080a409e16ffc1406"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:48.902030 25791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:26:48.972749 25791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:26:48.979041 25791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:26:48.979471 25791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:26:48.981024 25791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:26:48.983124 25791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:26:48.983229 25791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:48.983358 25791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:26:48.983424 25791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 20:26:49 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307208, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:26:49.184700 25791 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:35775
I20260809 20:26:49.184737 25932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:35775 every 8 connection(s)
I20260809 20:26:49.186196 25791 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:26:49.196305 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 25791
I20260809 20:26:49.196630 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:26:49.197696 12600 external_mini_cluster.cc:1428] Setting key 0b5e201168041bcb222d0d38f193ab13
Aug 09 20:26:49 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307208, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:26:49.262594 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:49.191489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:58195 (local address 127.12.78.62:40977)
0809 20:26:49.229300 (+ 37811us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:49.229311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:49.230678 (+ 1367us) server_negotiation.cc:407] Connection header received
0809 20:26:49.240245 (+ 9567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:49.240256 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:49.240398 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:49.240500 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:49.243585 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:49.244952 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:49.247932 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:49.248788 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:49.248918 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:49.254395 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:49.254405 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:49.254413 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:49.254467 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:49.258739 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:49.259741 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:49.259747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:49.259751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:49.259923 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:49.260445 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:49.260450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:49.260454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:49.260833 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:49.260952 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:49.261945 (+ 993us) server_negotiation.cc:299] Negotiation successful
0809 20:26:49.262205 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37568,"thread_start_us":37824,"threads_started":1}
I20260809 20:26:49.265184 25935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:49.190610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40977 (local address 127.12.78.1:58195)
0809 20:26:49.230301 (+ 39691us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:49.230396 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:49.238944 (+ 8548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:49.241219 (+ 2275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:49.241234 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:49.242411 (+ 1177us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:49.243342 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:49.243374 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:49.245232 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:49.245240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:49.247727 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:49.247748 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:49.249357 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:49.249367 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:49.249726 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:49.250730 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:49.250773 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:49.254044 (+ 3271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:49.258938 (+ 4894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:49.258950 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:49.258971 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:49.259593 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:49.260067 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:49.260072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:49.260076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:49.260325 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:49.261053 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:49.261061 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:49.261520 (+ 459us) client_negotiation.cc:769] Sending connection context
0809 20:26:49.263082 (+ 1562us) client_negotiation.cc:240] Negotiation successful
0809 20:26:49.263710 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39194,"thread_start_us":42044,"threads_started":1}
I20260809 20:26:49.267213 25933 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40977
I20260809 20:26:49.267557 25933 heartbeater.cc:464] Registering TS with master...
I20260809 20:26:49.268461 25933 heartbeater.cc:511] Master 127.12.78.62:40977 requested a full tablet report, sending...
I20260809 20:26:49.270876 25707 ts_manager.cc:194] Registered new tserver with Master: d42c4ce4de254a5382d23c5e01f7a513 (127.12.78.1:35775)
I20260809 20:26:49.272656 25707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:58195
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
W20260809 20:26:49.313216 12600 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.12.78.2: real 0.112s user 0.001s sys 0.023s
I20260809 20:26:49.314289 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.78.20:57479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40977
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 20:26:49 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307208, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
W20260809 20:26:49.780231 25939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:26:49.780622 25939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:26:49.780793 25939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:26:49.789644 25939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:26:49.789810 25939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:26:49.790007 25939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:26:49.806345 25939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.78.20:57479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:26:49.808161 25939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:26:49.809708 25939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:26:49.826252 25956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:49.826192 25957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:49.827072 25959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:49.827212 25939 server_base.cc:1034] running on GCE node
I20260809 20:26:49.827863 25939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:26:49.828894 25939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:26:49.830119 25939 hybrid_clock.cc:648] HybridClock initialized: now 1786307209830059 us; error 39 us; skew 500 ppm
I20260809 20:26:49.830425 25939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:26:49 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307209, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:26:49.839483 25939 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:26:49.841408 25939 webserver.cc:460] Webserver started at http://127.12.78.2:39003/ using document root <none> and password file <none>
I20260809 20:26:49.842180 25939 fs_manager.cc:359] Metadata directory not provided
I20260809 20:26:49.842324 25939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:26:49.842676 25939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 20:26:49 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307209, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:26:49.858748 25939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "c816db082dd64c86bad91cde1c3d46b0"
format_stamp: "Formatted at 2026-08-09 20:26:49 on dist-test-slave-rnrw"
server_key: "b888a2fd43f0af9bb80c4ae196fb7f4e"
server_key_iv: "2f79ee3cbdc7afca1b31141d3bde1889"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:49.859807 25939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "c816db082dd64c86bad91cde1c3d46b0"
format_stamp: "Formatted at 2026-08-09 20:26:49 on dist-test-slave-rnrw"
server_key: "b888a2fd43f0af9bb80c4ae196fb7f4e"
server_key_iv: "2f79ee3cbdc7afca1b31141d3bde1889"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:49.866040 25939 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 20:26:49.881975 25967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:49.883551 25939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:26:49.883713 25939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "c816db082dd64c86bad91cde1c3d46b0"
format_stamp: "Formatted at 2026-08-09 20:26:49 on dist-test-slave-rnrw"
server_key: "b888a2fd43f0af9bb80c4ae196fb7f4e"
server_key_iv: "2f79ee3cbdc7afca1b31141d3bde1889"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:49.883919 25939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:26:49.906575 25939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:26:49.911147 25939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:26:49.911486 25939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:26:49.912673 25939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:26:49.914297 25939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:26:49.914374 25939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:49.914526 25939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:26:49.914599 25939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:49.949816 25939 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:35435
I20260809 20:26:49.949870 26086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:35435 every 8 connection(s)
I20260809 20:26:49.951232 25939 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:26:49.956521 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 25939
I20260809 20:26:49.956751 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:26:49.957360 12600 external_mini_cluster.cc:1428] Setting key 92a288d769da85b1922660cbbcd15564
Aug 09 20:26:49 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307209, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:26:49.974773 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:49.953986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:39069 (local address 127.12.78.62:40977)
0809 20:26:49.954485 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:49.954490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:49.955344 (+ 854us) server_negotiation.cc:407] Connection header received
0809 20:26:49.958131 (+ 2787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:49.958139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:49.958267 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:49.958407 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:49.960257 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:49.961106 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:49.963660 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:49.964187 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:49.964287 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:49.968611 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:49.968616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:49.968620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:49.968654 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:49.971504 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:49.972295 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:49.972303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:49.972307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:49.972440 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:49.972958 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:49.972964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:49.972968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:49.973256 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:49.973415 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:49.974143 (+ 728us) server_negotiation.cc:299] Negotiation successful
0809 20:26:49.974380 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":274,"threads_started":1}
I20260809 20:26:49.975888 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:49.954393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40977 (local address 127.12.78.2:39069)
0809 20:26:49.955165 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:49.955211 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:49.957681 (+ 2470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:49.958697 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:49.958707 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:49.959178 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:49.960044 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:49.960073 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:49.961277 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:49.961285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:49.963491 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:49.963509 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:49.964439 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:49.964449 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:49.964846 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:49.965613 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:49.965657 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:49.968315 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:49.971677 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:49.971687 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:49.971704 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:49.972167 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:49.972607 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:49.972612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:49.972616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:49.972842 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:49.973391 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:49.973399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:49.973757 (+ 358us) client_negotiation.cc:769] Sending connection context
0809 20:26:49.974036 (+ 279us) client_negotiation.cc:240] Negotiation successful
0809 20:26:49.974597 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":323,"threads_started":1}
I20260809 20:26:49.977576 26087 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40977
I20260809 20:26:49.977927 26087 heartbeater.cc:464] Registering TS with master...
I20260809 20:26:49.978713 26087 heartbeater.cc:511] Master 127.12.78.62:40977 requested a full tablet report, sending...
I20260809 20:26:49.980376 25707 ts_manager.cc:194] Registered new tserver with Master: c816db082dd64c86bad91cde1c3d46b0 (127.12.78.2:35435)
I20260809 20:26:49.981299 25707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:39069
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:26:50.020164 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.78.20:57479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40977
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:26:50.277379 25933 heartbeater.cc:503] Master 127.12.78.62:40977 was elected leader, sending a full tablet report...
W20260809 20:26:50.388136 26094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:26:50.388466 26094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:26:50.388630 26094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:26:50.395923 26094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:26:50.396106 26094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:26:50.396256 26094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:26:50.406265 26094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.78.20:57479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:26:50.407547 26094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:26:50.408908 26094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:26:50.422839 26099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:50.422808 26102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:50.422885 26100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:50.423825 26094 server_base.cc:1034] running on GCE node
I20260809 20:26:50.424458 26094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:26:50.425483 26094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:26:50.426760 26094 hybrid_clock.cc:648] HybridClock initialized: now 1786307210426716 us; error 53 us; skew 500 ppm
I20260809 20:26:50.427096 26094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:26:50 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307210, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:26:50.430505 26094 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:26:50.432910 26094 webserver.cc:460] Webserver started at http://127.12.78.3:34763/ using document root <none> and password file <none>
I20260809 20:26:50.433653 26094 fs_manager.cc:359] Metadata directory not provided
I20260809 20:26:50.433772 26094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:26:50.434055 26094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 20:26:50 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307210, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:26:50.449497 26094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "9f2957a10aed40d1a575fd510ca0a61e"
format_stamp: "Formatted at 2026-08-09 20:26:50 on dist-test-slave-rnrw"
server_key: "25cc0e33249427c6b41114e8e340326a"
server_key_iv: "4e261a94078ca77a1bf8678218399cbc"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:50.450783 26094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "9f2957a10aed40d1a575fd510ca0a61e"
format_stamp: "Formatted at 2026-08-09 20:26:50 on dist-test-slave-rnrw"
server_key: "25cc0e33249427c6b41114e8e340326a"
server_key_iv: "4e261a94078ca77a1bf8678218399cbc"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:50.458309 26094 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.006s
I20260809 20:26:50.473671 26109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:50.475265 26094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 20:26:50.475454 26094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "9f2957a10aed40d1a575fd510ca0a61e"
format_stamp: "Formatted at 2026-08-09 20:26:50 on dist-test-slave-rnrw"
server_key: "25cc0e33249427c6b41114e8e340326a"
server_key_iv: "4e261a94078ca77a1bf8678218399cbc"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:50.475713 26094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:26:50.495782 26094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:26:50.500222 26094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:26:50.500617 26094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:26:50.501704 26094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:26:50.503173 26094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:26:50.503283 26094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:50.503444 26094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:26:50.503528 26094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:50.545078 26094 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:43465
I20260809 20:26:50.545171 26227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:43465 every 8 connection(s)
I20260809 20:26:50.546550 26094 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:26:50.549050 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 26094
I20260809 20:26:50.549266 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:26:50.549837 12600 external_mini_cluster.cc:1428] Setting key 0fe624190ebe0dec9e3b3ec2c96a1840
Aug 09 20:26:50 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307210, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:26:50.568250 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:50.549415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:36555 (local address 127.12.78.62:40977)
0809 20:26:50.550036 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:50.550045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:50.551226 (+ 1181us) server_negotiation.cc:407] Connection header received
0809 20:26:50.553271 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:50.553276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:50.553389 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:50.553509 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:50.554984 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:50.555695 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:50.557454 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:50.557903 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:50.558049 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:50.561714 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:50.561719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:50.561723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:50.561755 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:50.564750 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:50.565798 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:50.565804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:50.565807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:50.566006 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:50.566569 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:50.566573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:50.566575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:50.566860 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:50.566980 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:50.567776 (+ 796us) server_negotiation.cc:299] Negotiation successful
0809 20:26:50.567994 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":265,"threads_started":1}
I20260809 20:26:50.569315 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:50.550151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40977 (local address 127.12.78.3:36555)
0809 20:26:50.551069 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:50.551116 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:50.553012 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:50.553779 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:50.553792 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:50.554240 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:50.554818 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:50.554835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:50.555867 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:50.555870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:50.557268 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:50.557281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:50.558143 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:50.558171 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:50.558443 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:50.559115 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:50.559138 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:50.561438 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:50.564909 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:50.564921 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:50.564939 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:50.565666 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:50.566125 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:50.566132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:50.566136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:50.566423 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:50.566980 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:50.566987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:50.567363 (+ 376us) client_negotiation.cc:769] Sending connection context
0809 20:26:50.567627 (+ 264us) client_negotiation.cc:240] Negotiation successful
0809 20:26:50.568123 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":400,"threads_started":1}
I20260809 20:26:50.570673 26228 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40977
I20260809 20:26:50.570982 26228 heartbeater.cc:464] Registering TS with master...
I20260809 20:26:50.571679 26228 heartbeater.cc:511] Master 127.12.78.62:40977 requested a full tablet report, sending...
I20260809 20:26:50.573170 25707 ts_manager.cc:194] Registered new tserver with Master: 9f2957a10aed40d1a575fd510ca0a61e (127.12.78.3:43465)
I20260809 20:26:50.573956 25707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:36555
I20260809 20:26:50.578837 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:26:50.580750 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 25791
Aug 09 20:26:50 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:26:50.598774 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:50.581677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.12.78.1:35775)
0809 20:26:50.582319 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:50.582328 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:50.582361 (+ 33us) server_negotiation.cc:407] Connection header received
0809 20:26:50.582431 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:50.582447 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:50.582655 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:50.582827 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:50.584422 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:50.585520 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:50.587108 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:50.587759 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:50.587912 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:50.592531 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:50.592600 (+ 69us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:50.592624 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:50.592678 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:50.595634 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:50.596398 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:50.596408 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:50.596413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:50.596627 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:50.597040 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:50.597045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:50.597056 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:50.597419 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:50.597747 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:50.598109 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 20:26:50.598397 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":283,"threads_started":1}
I20260809 20:26:50.600114 25928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:26:50.684185 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 25791
I20260809 20:26:50.703123 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 25939
Aug 09 20:26:50 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:26:50.725214 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:50.704140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.12.78.2:35435)
0809 20:26:50.704730 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:50.704738 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:50.704762 (+ 24us) server_negotiation.cc:407] Connection header received
0809 20:26:50.704821 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:50.704828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:50.705041 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:50.705210 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:50.706538 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:50.707509 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:50.708961 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:50.709647 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:50.709798 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:50.719100 (+ 9302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:50.719113 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:50.719128 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:50.719176 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:50.722214 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:50.722952 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:50.722960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:50.722963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:50.723097 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:50.723481 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:50.723487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:50.723492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:50.723866 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:50.724202 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:50.724501 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 20:26:50.724826 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":265,"threads_started":1}
I20260809 20:26:50.726650 26082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:26:50.816948 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 25939
I20260809 20:26:50.836880 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 26094
Aug 09 20:26:50 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:26:50.854051 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:50.838018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.12.78.3:43465)
0809 20:26:50.838697 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:50.838704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:50.838734 (+ 30us) server_negotiation.cc:407] Connection header received
0809 20:26:50.838794 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:50.838800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:50.839021 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:50.839160 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:50.840421 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:50.841535 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:50.842746 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:50.843317 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:50.843414 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:50.848107 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:50.848120 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:50.848135 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:50.848174 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:50.851002 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:50.851750 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:50.851757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:50.851760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:50.851897 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:50.852299 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:50.852304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:50.852307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:50.852737 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:50.853100 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:50.853396 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 20:26:50.853670 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":374,"threads_started":1}
I20260809 20:26:50.855433 26223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:26:50.948498 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 26094
I20260809 20:26:50.968449 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 25673
I20260809 20:26:50.969889 25738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:26:51.077348 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 25673
I20260809 20:26:51.102854 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40977
--webserver_interface=127.12.78.62
--webserver_port=46403
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:40977
--ranger_config_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.78.20:57479/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.78.20:57479/kms with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:26:51.444260 26241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:26:51.444687 26241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:26:51.444857 26241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:26:51.452566 26241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:26:51.452685 26241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:26:51.452803 26241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:26:51.452908 26241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:26:51.452997 26241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:26:51.463521 26241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.78.20:57479/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:40977
--ranger_config_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=46403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:26:51.464975 26241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:26:51.466328 26241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:26:51.481050 26248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:51.481113 26250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:51.481019 26247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:51.481997 26241 server_base.cc:1034] running on GCE node
I20260809 20:26:51.482928 26241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:26:51.484679 26241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:26:51.485945 26241 hybrid_clock.cc:648] HybridClock initialized: now 1786307211485893 us; error 63 us; skew 500 ppm
I20260809 20:26:51.486363 26241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:26:51 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307211, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:26:51.489771 26241 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:26:51.491420 26241 webserver.cc:460] Webserver started at http://127.12.78.62:46403/ using document root <none> and password file <none>
I20260809 20:26:51.492142 26241 fs_manager.cc:359] Metadata directory not provided
I20260809 20:26:51.492305 26241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 20:26:51 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307211, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:26:51.574947 26241 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 20:26:51.578356 26259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:51.580209 26241 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 20:26:51.580412 26241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "a0653716cdf84d8c9f767c29eae93792"
format_stamp: "Formatted at 2026-08-09 20:26:47 on dist-test-slave-rnrw"
server_key: "82be749a0894083f5cc481b5ba21c556"
server_key_iv: "e65f154eb80240dd39eac147635fc480"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:51.581480 26241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:26:51.599356 26241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:26:51.603609 26241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:26:51.603921 26241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:26:51.623589 26241 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:40977
I20260809 20:26:51.623636 26314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:40977 every 8 connection(s)
I20260809 20:26:51.625248 26241 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:26:51.630712 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 26241
I20260809 20:26:51.631626 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.78.20:57479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:35775
--local_ip_for_outbound_sockets=127.12.78.1
--tserver_master_addrs=127.12.78.62:40977
--webserver_port=33039
--webserver_interface=127.12.78.1
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.78.20:57479/kms with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:26:51.632309 26315 sys_catalog.cc:263] Verifying existing consensus state
I20260809 20:26:51.635692 26315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792: Bootstrap starting.
I20260809 20:26:51.654675 26315 log.cc:824] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792: Log is configured to *not* fsync() on all Append() calls
I20260809 20:26:51.663875 26315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 20:26:51.664436 26315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792: Bootstrap complete.
I20260809 20:26:51.673923 26315 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0653716cdf84d8c9f767c29eae93792" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40977 } }
I20260809 20:26:51.675145 26315 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0653716cdf84d8c9f767c29eae93792, State: Initialized, Role: FOLLOWER
I20260809 20:26:51.675935 26315 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0653716cdf84d8c9f767c29eae93792" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40977 } }
I20260809 20:26:51.676208 26315 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:26:51.676337 26315 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:26:51.676528 26315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 1 FOLLOWER]: Advancing to term 2
I20260809 20:26:51.680488 26315 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0653716cdf84d8c9f767c29eae93792" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40977 } }
I20260809 20:26:51.681070 26315 leader_election.cc:302] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0653716cdf84d8c9f767c29eae93792; no voters:
I20260809 20:26:51.681741 26315 leader_election.cc:288] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 20:26:51.681880 26319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 20:26:51.682993 26319 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [term 2 LEADER]: Becoming Leader. State: Replica: a0653716cdf84d8c9f767c29eae93792, State: Running, Role: LEADER
I20260809 20:26:51.683566 26319 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0653716cdf84d8c9f767c29eae93792" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40977 } }
I20260809 20:26:51.683995 26315 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:26:51.691645 26321 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0653716cdf84d8c9f767c29eae93792. Latest consensus state: current_term: 2 leader_uuid: "a0653716cdf84d8c9f767c29eae93792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0653716cdf84d8c9f767c29eae93792" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40977 } } }
I20260809 20:26:51.692155 26321 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [sys.catalog]: This master's current role is: LEADER
I20260809 20:26:51.692809 26320 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a0653716cdf84d8c9f767c29eae93792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0653716cdf84d8c9f767c29eae93792" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40977 } } }
I20260809 20:26:51.693162 26320 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792 [sys.catalog]: This master's current role is: LEADER
I20260809 20:26:51.696213 26327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:26:51.703538 26327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:26:51.705402 26327 catalog_manager.cc:1259] Loaded cluster ID: a614185df1af45ca93a19e7700d5c5f2
I20260809 20:26:51.705521 26327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:26:51.710765 26327 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:26:51.712683 26327 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792: Loaded TSK: 0
I20260809 20:26:51.714627 26327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:26:51.719740 26315 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260809 20:26:52.107405 26317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:26:52.107877 26317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:26:52.108060 26317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:26:52.120157 26317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:26:52.120324 26317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:26:52.120528 26317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:26:52.137162 26317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.78.20:57479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:35775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=33039
--tserver_master_addrs=127.12.78.62:40977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:26:52.139022 26317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:26:52.140859 26317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:26:52.178218 26359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:52.178201 26358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:52.188480 26317 server_base.cc:1034] running on GCE node
W20260809 20:26:52.189667 26361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:52.190619 26317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:26:52.198137 26317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:26:52.199491 26317 hybrid_clock.cc:648] HybridClock initialized: now 1786307212199411 us; error 78 us; skew 500 ppm
I20260809 20:26:52.199932 26317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:26:52 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:26:52.206846 26317 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:26:52.221645 26317 webserver.cc:460] Webserver started at http://127.12.78.1:33039/ using document root <none> and password file <none>
I20260809 20:26:52.222723 26317 fs_manager.cc:359] Metadata directory not provided
I20260809 20:26:52.222913 26317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 20:26:52 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:26:52.290746 26317 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.006s
I20260809 20:26:52.296838 26370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:52.298904 26317 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.006s
I20260809 20:26:52.299121 26317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "d42c4ce4de254a5382d23c5e01f7a513"
format_stamp: "Formatted at 2026-08-09 20:26:48 on dist-test-slave-rnrw"
server_key: "21740a3b422e31e108072712dbb98139"
server_key_iv: "9e5c36079ce4685080a409e16ffc1406"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:52.300571 26317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:26:52.346522 26317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:26:52.351974 26317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:26:52.352346 26317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:26:52.353725 26317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:26:52.355675 26317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:26:52.355791 26317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:52.355901 26317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:26:52.355983 26317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:52.399327 26317 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:35775
I20260809 20:26:52.399499 26489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:35775 every 8 connection(s)
I20260809 20:26:52.401283 26317 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:26:52.410985 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 26317
I20260809 20:26:52.411832 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.78.20:57479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:35435
--local_ip_for_outbound_sockets=127.12.78.2
--tserver_master_addrs=127.12.78.62:40977
--webserver_port=39003
--webserver_interface=127.12.78.2
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.78.20:57479/kms with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 20:26:52 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:26:52.452280 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:52.405698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:51235 (local address 127.12.78.62:40977)
0809 20:26:52.416740 (+ 11042us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:52.416761 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:52.420680 (+ 3919us) server_negotiation.cc:407] Connection header received
0809 20:26:52.424567 (+ 3887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:52.424602 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:52.425198 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:52.425733 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:52.428786 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:52.431128 (+ 2342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:52.434214 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:52.435571 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:52.435791 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:52.440997 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:52.441012 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:52.441035 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:52.441107 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:52.445731 (+ 4624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:52.446929 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:52.446941 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:52.446950 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:52.447210 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:52.447926 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:52.447933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:52.447937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:52.448741 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:52.448975 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:52.450198 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0809 20:26:52.450712 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10670,"thread_start_us":10366,"threads_started":1}
I20260809 20:26:52.454020 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:52.405057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40977 (local address 127.12.78.1:51235)
0809 20:26:52.420394 (+ 15337us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:52.420445 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:52.423629 (+ 3184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:52.426700 (+ 3071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:52.426714 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:52.427518 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:52.428445 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:52.428478 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:52.431448 (+ 2970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:52.431454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:52.433975 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:52.434011 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:52.436314 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:52.436324 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:52.436712 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:52.437596 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:52.437639 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:52.440602 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:52.445967 (+ 5365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:52.445979 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:52.446000 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:52.446720 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:52.447420 (+ 700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:52.447426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:52.447430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:52.447764 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:52.449087 (+ 1323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:52.449097 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:52.449544 (+ 447us) client_negotiation.cc:769] Sending connection context
0809 20:26:52.452147 (+ 2603us) client_negotiation.cc:240] Negotiation successful
0809 20:26:52.452866 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14962,"thread_start_us":15294,"threads_started":1}
I20260809 20:26:52.458138 26490 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40977
I20260809 20:26:52.458567 26490 heartbeater.cc:464] Registering TS with master...
I20260809 20:26:52.459563 26490 heartbeater.cc:511] Master 127.12.78.62:40977 requested a full tablet report, sending...
W20260809 20:26:52.463565 26490 heartbeater.cc:644] Failed to heartbeat to 127.12.78.62:40977 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 20:26:52.463881 26490 heartbeater.cc:464] Registering TS with master...
I20260809 20:26:52.464203 26490 heartbeater.cc:511] Master 127.12.78.62:40977 requested a full tablet report, sending...
I20260809 20:26:52.465525 26490 heartbeater.cc:464] Registering TS with master...
I20260809 20:26:52.465824 26490 heartbeater.cc:511] Master 127.12.78.62:40977 requested a full tablet report, sending...
W20260809 20:26:52.467011 26490 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 20:26:52.861652 26494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:26:52.862076 26494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:26:52.862239 26494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:26:52.869997 26494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:26:52.870175 26494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:26:52.870352 26494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:26:52.887228 26494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.78.20:57479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:35435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=39003
--tserver_master_addrs=127.12.78.62:40977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:26:52.889181 26494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:26:52.891000 26494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:26:52.920758 26510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:52.921556 26494 server_base.cc:1034] running on GCE node
W20260809 20:26:52.921641 26509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:52.921756 26512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:52.922590 26494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:26:52.923915 26494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:26:52.925154 26494 hybrid_clock.cc:648] HybridClock initialized: now 1786307212925059 us; error 109 us; skew 500 ppm
I20260809 20:26:52.925529 26494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:26:52 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:26:52.929867 26494 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:26:52.954399 26494 webserver.cc:460] Webserver started at http://127.12.78.2:39003/ using document root <none> and password file <none>
I20260809 20:26:52.955492 26494 fs_manager.cc:359] Metadata directory not provided
I20260809 20:26:52.955734 26494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 20:26:52 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:26:52.988094 26494 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.005s sys 0.004s
I20260809 20:26:52.992328 26522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:52.994452 26494 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 20:26:52.994688 26494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "c816db082dd64c86bad91cde1c3d46b0"
format_stamp: "Formatted at 2026-08-09 20:26:49 on dist-test-slave-rnrw"
server_key: "b888a2fd43f0af9bb80c4ae196fb7f4e"
server_key_iv: "2f79ee3cbdc7afca1b31141d3bde1889"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:52.996279 26494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 09 20:26:53 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786307213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:26:53.014984 26494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:26:53.021876 26494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:26:53.022236 26494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:26:53.029692 26494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:26:53.031710 26494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:26:53.031854 26494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:53.031984 26494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:26:53.032076 26494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:53.094945 26494 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:35435
I20260809 20:26:53.094993 26642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:35435 every 8 connection(s)
I20260809 20:26:53.096652 26494 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:26:53.101814 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 26494
I20260809 20:26:53.102931 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.78.20:57479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:43465
--local_ip_for_outbound_sockets=127.12.78.3
--tserver_master_addrs=127.12.78.62:40977
--webserver_port=34763
--webserver_interface=127.12.78.3
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.78.20:57479/kms with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 20:26:53 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:26:53.152940 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:53.100979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:43765 (local address 127.12.78.62:40977)
0809 20:26:53.131781 (+ 30802us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:53.131788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:53.131810 (+ 22us) server_negotiation.cc:407] Connection header received
0809 20:26:53.131876 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:53.131882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:53.131964 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:53.132082 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:53.134445 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:53.135715 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:53.138460 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:53.139257 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:53.139399 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:53.144351 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:53.144360 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:53.144368 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:53.144416 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:53.148640 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:53.149792 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:53.149798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:53.149802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:53.149986 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:53.150623 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:53.150630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:53.150636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:53.151039 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:53.151216 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:53.152254 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0809 20:26:53.152497 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30646,"thread_start_us":30511,"threads_started":1}
I20260809 20:26:53.154819 26646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:53.105699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40977 (local address 127.12.78.2:43765)
0809 20:26:53.128242 (+ 22543us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:53.128310 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:53.130996 (+ 2686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:53.132452 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:53.132469 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:53.133255 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:53.134236 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:53.134268 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:53.135926 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:53.135932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:53.138248 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:53.138279 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:53.139749 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:53.139758 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:53.140077 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:53.140936 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:53.140973 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:53.144061 (+ 3088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:53.148842 (+ 4781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:53.148856 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:53.148878 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:53.149610 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:53.150129 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:53.150137 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:53.150144 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:53.150454 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:53.151212 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:53.151223 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:53.151732 (+ 509us) client_negotiation.cc:769] Sending connection context
0809 20:26:53.152090 (+ 358us) client_negotiation.cc:240] Negotiation successful
0809 20:26:53.152799 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22136,"thread_start_us":21784,"threads_started":1}
I20260809 20:26:53.156725 26643 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40977
I20260809 20:26:53.157099 26643 heartbeater.cc:464] Registering TS with master...
I20260809 20:26:53.158053 26643 heartbeater.cc:511] Master 127.12.78.62:40977 requested a full tablet report, sending...
I20260809 20:26:53.160485 26278 ts_manager.cc:194] Registered new tserver with Master: c816db082dd64c86bad91cde1c3d46b0 (127.12.78.2:35435)
I20260809 20:26:53.162807 26278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:43765
I20260809 20:26:53.467209 26490 heartbeater.cc:464] Registering TS with master...
I20260809 20:26:53.467762 26490 heartbeater.cc:511] Master 127.12.78.62:40977 requested a full tablet report, sending...
I20260809 20:26:53.469381 26278 ts_manager.cc:194] Registered new tserver with Master: d42c4ce4de254a5382d23c5e01f7a513 (127.12.78.1:35775)
I20260809 20:26:53.470824 26278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:51235
Aug 09 20:26:53 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
W20260809 20:26:53.626583 26647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:26:53.627020 26647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:26:53.627197 26647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:26:53.638919 26647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:26:53.639107 26647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:26:53.639321 26647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:26:53.655763 26647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.78.20:57479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:43465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=34763
--tserver_master_addrs=127.12.78.62:40977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:26:53.657742 26647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:26:53.659773 26647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:26:53.678884 26654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:53.678912 26653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:53.685438 26647 server_base.cc:1034] running on GCE node
W20260809 20:26:53.685755 26656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:53.686779 26647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:26:53.688113 26647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:26:53.689353 26647 hybrid_clock.cc:648] HybridClock initialized: now 1786307213689266 us; error 80 us; skew 500 ppm
I20260809 20:26:53.689740 26647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:26:53 dist-test-slave-rnrw krb5kdc[25201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:26:53.694543 26647 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:26:53.696722 26647 webserver.cc:460] Webserver started at http://127.12.78.3:34763/ using document root <none> and password file <none>
I20260809 20:26:53.697706 26647 fs_manager.cc:359] Metadata directory not provided
I20260809 20:26:53.697871 26647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 20:26:53 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:26:53.725286 26647 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.007s sys 0.000s
I20260809 20:26:53.729736 26665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:53.731782 26647 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 20:26:53.731979 26647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "9f2957a10aed40d1a575fd510ca0a61e"
format_stamp: "Formatted at 2026-08-09 20:26:50 on dist-test-slave-rnrw"
server_key: "25cc0e33249427c6b41114e8e340326a"
server_key_iv: "4e261a94078ca77a1bf8678218399cbc"
server_key_version: "kuduclusterkey@0"
I20260809 20:26:53.733435 26647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:26:53.754241 26647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:26:53.759661 26647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:26:53.760002 26647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:26:53.761399 26647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:26:53.763600 26647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:26:53.763742 26647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:53.763880 26647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:26:53.763976 26647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 20:26:53 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for HTTP/127.12.78.20@KRBTEST.COM
I20260809 20:26:54.023872 26647 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:43465
I20260809 20:26:54.026110 26647 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:26:54.023638 26787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:43465 every 8 connection(s)
I20260809 20:26:54.032658 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 26647
I20260809 20:26:54.101876 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.034754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.12.78.62:40977)
0809 20:26:54.070979 (+ 36225us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:54.070986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:54.071490 (+ 504us) server_negotiation.cc:407] Connection header received
0809 20:26:54.071593 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:54.071599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:54.071736 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:54.071923 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:54.074768 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.076034 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.078457 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.079128 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.079269 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.083845 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:54.083852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:54.083857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:54.083903 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:54.088238 (+ 4335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:54.090163 (+ 1925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:54.090170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:54.090173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:54.090341 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:54.091669 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:54.091674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:54.091678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:54.092014 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:54.092299 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:54.093422 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0809 20:26:54.093686 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36076,"thread_start_us":35007,"threads_started":1}
Aug 09 20:26:54 dist-test-slave-rnrw krb5kdc[25201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:26:54.120651 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.034412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:54827 (local address 127.12.78.62:40977)
0809 20:26:54.069053 (+ 34641us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:54.069060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:54.074659 (+ 5599us) server_negotiation.cc:407] Connection header received
0809 20:26:54.079680 (+ 5021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:54.079686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:54.079780 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:54.079861 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:54.082331 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.083614 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.090522 (+ 6908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.091210 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.091447 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.107777 (+ 16330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:54.107785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:54.107790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:54.107841 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:54.111422 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:54.112603 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:54.112610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:54.112613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:54.112822 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:54.113372 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:54.113378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:54.113381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:54.113769 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:54.113914 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:54.119921 (+ 6007us) server_negotiation.cc:299] Negotiation successful
0809 20:26:54.120179 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34481,"thread_start_us":36970,"threads_started":1}
I20260809 20:26:54.121752 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.033787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40977 (local address 127.12.78.3:54827)
0809 20:26:54.072683 (+ 38896us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:54.072753 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:54.078193 (+ 5440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:54.080518 (+ 2325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:54.080532 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:54.081231 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:54.082080 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:54.082108 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.084859 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.084864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:54.089987 (+ 5123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:54.090006 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.092584 (+ 2578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.092593 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:54.092901 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.103151 (+ 10250us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:54.103189 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:54.107411 (+ 4222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:54.111636 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:54.111648 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:54.111668 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:54.112406 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:54.112981 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:54.112986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:54.112990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:54.113231 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:54.114035 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:54.114044 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:54.114486 (+ 442us) client_negotiation.cc:769] Sending connection context
0809 20:26:54.119803 (+ 5317us) client_negotiation.cc:240] Negotiation successful
0809 20:26:54.120465 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38503,"thread_start_us":42310,"threads_started":1}
I20260809 20:26:54.123208 26788 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40977
I20260809 20:26:54.123524 26788 heartbeater.cc:464] Registering TS with master...
I20260809 20:26:54.124321 26788 heartbeater.cc:511] Master 127.12.78.62:40977 requested a full tablet report, sending...
I20260809 20:26:54.125905 26278 ts_manager.cc:194] Registered new tserver with Master: 9f2957a10aed40d1a575fd510ca0a61e (127.12.78.3:43465)
I20260809 20:26:54.126907 26278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:54827
I20260809 20:26:54.131480 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:26:54.154994 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.138668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.12.78.62:40977)
0809 20:26:54.139114 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:54.139122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:54.139412 (+ 290us) server_negotiation.cc:407] Connection header received
0809 20:26:54.139626 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:54.139631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:54.139710 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:54.139822 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:54.141750 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.142461 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.143924 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.144356 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.144494 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.147748 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:54.147756 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:54.147761 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:54.147807 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:54.151563 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:54.152457 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:54.152464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:54.152468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:54.152672 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:54.153153 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:54.153158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:54.153161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:54.153551 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:54.153780 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:54.154380 (+ 600us) server_negotiation.cc:299] Negotiation successful
0809 20:26:54.154642 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260809 20:26:54.167920 26643 heartbeater.cc:503] Master 127.12.78.62:40977 was elected leader, sending a full tablet report...
I20260809 20:26:54.162690 26278 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:26:54.175254 26278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:26:54.218273 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.194029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:43465 (local address 127.0.0.1:38682)
0809 20:26:54.195981 (+ 1952us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:54.196011 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:54.196163 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:54.208303 (+ 12140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:54.208314 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:54.208360 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:26:54.209002 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:54.209020 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.212035 (+ 3015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.212040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:54.215209 (+ 3169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:54.215223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.217251 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.217257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:54.217474 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.217531 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 20:26:54.217669 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 20:26:54.217880 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1786,"thread_start_us":21518,"threads_started":1}
I20260809 20:26:54.218278 26814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.194194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.12.78.3:43465)
0809 20:26:54.205982 (+ 11788us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:54.205994 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:54.206039 (+ 45us) server_negotiation.cc:407] Connection header received
0809 20:26:54.206190 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:54.206203 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:54.206617 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:54.209300 (+ 2683us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:26:54.209699 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.211845 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.215401 (+ 3556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.216954 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.217127 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.217393 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:54.217753 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 20:26:54.217933 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11457,"thread_start_us":11236,"threads_started":1}
I20260809 20:26:54.220498 26811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.193244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:35435 (local address 127.0.0.1:42018)
0809 20:26:54.194062 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:54.194136 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:54.194349 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:54.212159 (+ 17810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:54.212170 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:54.212214 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:26:54.212784 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:54.212800 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.215738 (+ 2938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.215748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:54.217902 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:54.217915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.219746 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.219755 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:54.219916 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.219950 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 20:26:54.220009 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 20:26:54.220114 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":2,"thread_start_us":3581,"threads_started":1}
I20260809 20:26:54.221772 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.193673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.12.78.2:35435)
0809 20:26:54.207486 (+ 13813us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:54.207497 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:54.207534 (+ 37us) server_negotiation.cc:407] Connection header received
0809 20:26:54.207659 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:54.207671 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:54.207948 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:54.213087 (+ 5139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:26:54.213409 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.215541 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.218094 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.219495 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.220825 (+ 1330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.221163 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:54.221238 (+ 75us) server_negotiation.cc:299] Negotiation successful
0809 20:26:54.221408 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13558,"thread_start_us":13366,"threads_started":1}
I20260809 20:26:54.223270 26718 tablet_service.cc:1467] Processing CreateTablet for tablet d716c80d604b4c07b30d5238aa7d5e12 (DEFAULT_TABLE table=test-table [id=c18f0b961dbc4ce7bf07128deb36fe08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:26:54.224105 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.195306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:35775 (local address 127.0.0.1:54706)
0809 20:26:54.215889 (+ 20583us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:54.215914 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:54.216058 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:54.216738 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:54.216743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:54.216768 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:26:54.217258 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:54.217271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.219284 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.219289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:54.221422 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:54.221436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.223219 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.223226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:54.223416 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.223450 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 20:26:54.223570 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 20:26:54.223704 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20435,"thread_start_us":3325,"threads_started":1}
I20260809 20:26:54.224154 26816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.195909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.12.78.1:35775)
0809 20:26:54.205227 (+ 9318us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:54.205241 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:54.216054 (+ 10813us) server_negotiation.cc:407] Connection header received
0809 20:26:54.216211 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:54.216219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:54.216492 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:54.216750 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:26:54.217520 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.219108 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.221609 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.222998 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.223159 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.223376 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:54.223615 (+ 239us) server_negotiation.cc:299] Negotiation successful
0809 20:26:54.223772 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9070,"thread_start_us":8861,"threads_started":1}
I20260809 20:26:54.224921 26718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d716c80d604b4c07b30d5238aa7d5e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:26:54.225395 26575 tablet_service.cc:1467] Processing CreateTablet for tablet d716c80d604b4c07b30d5238aa7d5e12 (DEFAULT_TABLE table=test-table [id=c18f0b961dbc4ce7bf07128deb36fe08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:26:54.226801 26575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d716c80d604b4c07b30d5238aa7d5e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:26:54.227401 26421 tablet_service.cc:1467] Processing CreateTablet for tablet d716c80d604b4c07b30d5238aa7d5e12 (DEFAULT_TABLE table=test-table [id=c18f0b961dbc4ce7bf07128deb36fe08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:26:54.228802 26421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d716c80d604b4c07b30d5238aa7d5e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:26:54.240624 26817 tablet_bootstrap.cc:492] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513: Bootstrap starting.
I20260809 20:26:54.242283 26820 tablet_bootstrap.cc:492] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0: Bootstrap starting.
I20260809 20:26:54.243980 26818 tablet_bootstrap.cc:492] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e: Bootstrap starting.
I20260809 20:26:54.245478 26817 tablet_bootstrap.cc:654] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513: Neither blocks nor log segments found. Creating new log.
I20260809 20:26:54.245910 26820 tablet_bootstrap.cc:654] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0: Neither blocks nor log segments found. Creating new log.
I20260809 20:26:54.246928 26818 tablet_bootstrap.cc:654] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e: Neither blocks nor log segments found. Creating new log.
I20260809 20:26:54.247123 26820 log.cc:824] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0: Log is configured to *not* fsync() on all Append() calls
I20260809 20:26:54.247679 26817 log.cc:824] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513: Log is configured to *not* fsync() on all Append() calls
I20260809 20:26:54.248246 26818 log.cc:824] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e: Log is configured to *not* fsync() on all Append() calls
I20260809 20:26:54.250269 26820 tablet_bootstrap.cc:492] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0: No bootstrap required, opened a new log
I20260809 20:26:54.250557 26820 ts_tablet_manager.cc:1397] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 20:26:54.251262 26817 tablet_bootstrap.cc:492] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513: No bootstrap required, opened a new log
I20260809 20:26:54.251612 26817 ts_tablet_manager.cc:1397] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260809 20:26:54.258505 26820 raft_consensus.cc:348] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c816db082dd64c86bad91cde1c3d46b0" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 35435 } } peers { permanent_uuid: "d42c4ce4de254a5382d23c5e01f7a513" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35775 } } peers { permanent_uuid: "9f2957a10aed40d1a575fd510ca0a61e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43465 } }
I20260809 20:26:54.258903 26820 raft_consensus.cc:374] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:26:54.259012 26820 raft_consensus.cc:729] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c816db082dd64c86bad91cde1c3d46b0, State: Initialized, Role: FOLLOWER
I20260809 20:26:54.259701 26820 consensus_queue.cc:260] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c816db082dd64c86bad91cde1c3d46b0" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 35435 } } peers { permanent_uuid: "d42c4ce4de254a5382d23c5e01f7a513" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35775 } } peers { permanent_uuid: "9f2957a10aed40d1a575fd510ca0a61e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43465 } }
I20260809 20:26:54.259940 26818 tablet_bootstrap.cc:492] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e: No bootstrap required, opened a new log
I20260809 20:26:54.260255 26818 ts_tablet_manager.cc:1397] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20260809 20:26:54.262382 26820 ts_tablet_manager.cc:1428] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260809 20:26:54.263262 26817 raft_consensus.cc:348] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c816db082dd64c86bad91cde1c3d46b0" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 35435 } } peers { permanent_uuid: "d42c4ce4de254a5382d23c5e01f7a513" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35775 } } peers { permanent_uuid: "9f2957a10aed40d1a575fd510ca0a61e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43465 } }
I20260809 20:26:54.263722 26817 raft_consensus.cc:374] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:26:54.263825 26817 raft_consensus.cc:729] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42c4ce4de254a5382d23c5e01f7a513, State: Initialized, Role: FOLLOWER
I20260809 20:26:54.264448 26817 consensus_queue.cc:260] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c816db082dd64c86bad91cde1c3d46b0" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 35435 } } peers { permanent_uuid: "d42c4ce4de254a5382d23c5e01f7a513" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35775 } } peers { permanent_uuid: "9f2957a10aed40d1a575fd510ca0a61e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43465 } }
I20260809 20:26:54.265672 26490 heartbeater.cc:503] Master 127.12.78.62:40977 was elected leader, sending a full tablet report...
I20260809 20:26:54.266291 26817 ts_tablet_manager.cc:1428] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260809 20:26:54.271297 26818 raft_consensus.cc:348] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c816db082dd64c86bad91cde1c3d46b0" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 35435 } } peers { permanent_uuid: "d42c4ce4de254a5382d23c5e01f7a513" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35775 } } peers { permanent_uuid: "9f2957a10aed40d1a575fd510ca0a61e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43465 } }
I20260809 20:26:54.271936 26818 raft_consensus.cc:374] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:26:54.272111 26818 raft_consensus.cc:729] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2957a10aed40d1a575fd510ca0a61e, State: Initialized, Role: FOLLOWER
I20260809 20:26:54.272775 26818 consensus_queue.cc:260] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c816db082dd64c86bad91cde1c3d46b0" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 35435 } } peers { permanent_uuid: "d42c4ce4de254a5382d23c5e01f7a513" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35775 } } peers { permanent_uuid: "9f2957a10aed40d1a575fd510ca0a61e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43465 } }
I20260809 20:26:54.273823 26788 heartbeater.cc:503] Master 127.12.78.62:40977 was elected leader, sending a full tablet report...
I20260809 20:26:54.274308 26818 ts_tablet_manager.cc:1428] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
W20260809 20:26:54.320255 26793 tablet.cc:2369] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:26:54.343335 26826 raft_consensus.cc:482] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:26:54.343616 26826 raft_consensus.cc:504] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c816db082dd64c86bad91cde1c3d46b0" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 35435 } } peers { permanent_uuid: "d42c4ce4de254a5382d23c5e01f7a513" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35775 } } peers { permanent_uuid: "9f2957a10aed40d1a575fd510ca0a61e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43465 } }
I20260809 20:26:54.345399 26826 leader_election.cc:288] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c816db082dd64c86bad91cde1c3d46b0 (127.12.78.2:35435), d42c4ce4de254a5382d23c5e01f7a513 (127.12.78.1:35775)
I20260809 20:26:54.352659 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.345753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:35775 (local address 127.12.78.3:41775)
0809 20:26:54.346111 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:54.346135 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:54.346287 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:54.346743 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:54.346748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:54.346778 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:26:54.347000 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:54.347008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.348334 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.348338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:54.350365 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:54.350379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.351916 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.351920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:54.352065 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.352090 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 20:26:54.352168 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 20:26:54.352295 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260809 20:26:54.352689 26816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.345879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:41775 (local address 127.12.78.1:35775)
0809 20:26:54.346154 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:54.346164 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:54.346203 (+ 39us) server_negotiation.cc:407] Connection header received
0809 20:26:54.346487 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:54.346493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:54.346590 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:54.346684 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:26:54.347148 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.348194 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.350573 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.351772 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.351908 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.352013 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:54.352231 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 20:26:54.352375 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 20:26:54.353183 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.345975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:34055 (local address 127.12.78.2:35435)
0809 20:26:54.346172 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:54.346180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:54.346787 (+ 607us) server_negotiation.cc:407] Connection header received
0809 20:26:54.346941 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:54.346945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:54.347018 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:54.347137 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:26:54.347651 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.348646 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.350946 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.352193 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.352332 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.352445 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:54.352845 (+ 400us) server_negotiation.cc:299] Negotiation successful
0809 20:26:54.352960 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 20:26:54.353760 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.345796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:35435 (local address 127.12.78.3:34055)
0809 20:26:54.346642 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:54.346681 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:54.346805 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:54.347172 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:54.347175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:54.347194 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:26:54.347477 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:54.347484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.348785 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.348790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:54.350780 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:54.350793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.352421 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.352430 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:54.352631 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.352675 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 20:26:54.353370 (+ 695us) client_negotiation.cc:240] Negotiation successful
0809 20:26:54.353493 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"spinlock_wait_cycles":6528,"thread_start_us":577,"threads_started":1}
I20260809 20:26:54.353914 26443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d716c80d604b4c07b30d5238aa7d5e12" candidate_uuid: "9f2957a10aed40d1a575fd510ca0a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42c4ce4de254a5382d23c5e01f7a513" is_pre_election: true
I20260809 20:26:54.354554 26443 raft_consensus.cc:2454] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f2957a10aed40d1a575fd510ca0a61e in term 0.
I20260809 20:26:54.354815 26596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d716c80d604b4c07b30d5238aa7d5e12" candidate_uuid: "9f2957a10aed40d1a575fd510ca0a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c816db082dd64c86bad91cde1c3d46b0" is_pre_election: true
I20260809 20:26:54.355345 26667 leader_election.cc:302] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f2957a10aed40d1a575fd510ca0a61e, d42c4ce4de254a5382d23c5e01f7a513; no voters:
I20260809 20:26:54.355445 26596 raft_consensus.cc:2454] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f2957a10aed40d1a575fd510ca0a61e in term 0.
I20260809 20:26:54.355810 26826 raft_consensus.cc:2781] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:26:54.356019 26826 raft_consensus.cc:482] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:26:54.356155 26826 raft_consensus.cc:3037] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:26:54.358647 26826 raft_consensus.cc:504] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c816db082dd64c86bad91cde1c3d46b0" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 35435 } } peers { permanent_uuid: "d42c4ce4de254a5382d23c5e01f7a513" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35775 } } peers { permanent_uuid: "9f2957a10aed40d1a575fd510ca0a61e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43465 } }
I20260809 20:26:54.359264 26826 leader_election.cc:288] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [CANDIDATE]: Term 1 election: Requested vote from peers c816db082dd64c86bad91cde1c3d46b0 (127.12.78.2:35435), d42c4ce4de254a5382d23c5e01f7a513 (127.12.78.1:35775)
I20260809 20:26:54.359681 26596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d716c80d604b4c07b30d5238aa7d5e12" candidate_uuid: "9f2957a10aed40d1a575fd510ca0a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c816db082dd64c86bad91cde1c3d46b0"
I20260809 20:26:54.359761 26443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d716c80d604b4c07b30d5238aa7d5e12" candidate_uuid: "9f2957a10aed40d1a575fd510ca0a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42c4ce4de254a5382d23c5e01f7a513"
I20260809 20:26:54.359884 26596 raft_consensus.cc:3037] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:26:54.360009 26443 raft_consensus.cc:3037] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:26:54.362818 26596 raft_consensus.cc:2454] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f2957a10aed40d1a575fd510ca0a61e in term 1.
I20260809 20:26:54.362898 26443 raft_consensus.cc:2454] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f2957a10aed40d1a575fd510ca0a61e in term 1.
I20260809 20:26:54.363481 26667 leader_election.cc:302] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f2957a10aed40d1a575fd510ca0a61e, d42c4ce4de254a5382d23c5e01f7a513; no voters:
I20260809 20:26:54.363821 26826 raft_consensus.cc:2781] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:26:54.364445 26826 raft_consensus.cc:686] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [term 1 LEADER]: Becoming Leader. State: Replica: 9f2957a10aed40d1a575fd510ca0a61e, State: Running, Role: LEADER
I20260809 20:26:54.365193 26826 consensus_queue.cc:237] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c816db082dd64c86bad91cde1c3d46b0" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 35435 } } peers { permanent_uuid: "d42c4ce4de254a5382d23c5e01f7a513" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35775 } } peers { permanent_uuid: "9f2957a10aed40d1a575fd510ca0a61e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43465 } }
I20260809 20:26:54.371376 26278 catalog_manager.cc:5495] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f2957a10aed40d1a575fd510ca0a61e (127.12.78.3). New cstate: current_term: 1 leader_uuid: "9f2957a10aed40d1a575fd510ca0a61e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c816db082dd64c86bad91cde1c3d46b0" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 35435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d42c4ce4de254a5382d23c5e01f7a513" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f2957a10aed40d1a575fd510ca0a61e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43465 } health_report { overall_health: HEALTHY } } }
W20260809 20:26:54.384829 26644 tablet.cc:2369] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:26:54.403815 26814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.397869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.12.78.3:43465)
0809 20:26:54.398083 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:54.398088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:54.398106 (+ 18us) server_negotiation.cc:407] Connection header received
0809 20:26:54.398402 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:54.398406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:54.398498 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:54.398661 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:26:54.399073 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.399873 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.401318 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.401855 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.402012 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.402583 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:26:54.403227 (+ 644us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:26:54.403390 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:54.403454 (+ 64us) server_negotiation.cc:299] Negotiation successful
0809 20:26:54.403541 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260809 20:26:54.410435 26491 tablet.cc:2369] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:26:54.413172 26443 raft_consensus.cc:1261] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513 [term 1 FOLLOWER]: Refusing update from remote peer 9f2957a10aed40d1a575fd510ca0a61e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:26:54.413338 26596 raft_consensus.cc:1261] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0 [term 1 FOLLOWER]: Refusing update from remote peer 9f2957a10aed40d1a575fd510ca0a61e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:26:54.414273 26826 consensus_queue.cc:1035] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42c4ce4de254a5382d23c5e01f7a513" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:26:54.414572 26828 consensus_queue.cc:1035] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c816db082dd64c86bad91cde1c3d46b0" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 35435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:26:54.433087 26834 mvcc.cc:205] Tried to move back new op lower bound from 7316714350225440768 to 7316714350045097984. Current Snapshot: MvccSnapshot[applied={T|T < 7316714350225440768}]
I20260809 20:26:54.442452 26278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:26:54.442741 26278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:26:54.446295 26278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0653716cdf84d8c9f767c29eae93792: Sending DeleteTablet for 3 replicas of tablet d716c80d604b4c07b30d5238aa7d5e12
I20260809 20:26:54.447571 26575 tablet_service.cc:1514] Processing DeleteTablet for tablet d716c80d604b4c07b30d5238aa7d5e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:26:54 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:42018
I20260809 20:26:54.447642 26718 tablet_service.cc:1514] Processing DeleteTablet for tablet d716c80d604b4c07b30d5238aa7d5e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:26:54 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:38682
I20260809 20:26:54.447629 26421 tablet_service.cc:1514] Processing DeleteTablet for tablet d716c80d604b4c07b30d5238aa7d5e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:26:54 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:54706
I20260809 20:26:54.459949 26844 tablet_replica.cc:330] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513: stopping tablet replica
I20260809 20:26:54.460585 26844 raft_consensus.cc:2229] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:26:54.461244 26844 raft_consensus.cc:2258] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:26:54.465116 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 26317
I20260809 20:26:54.467801 26842 tablet_replica.cc:330] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0: stopping tablet replica
I20260809 20:26:54.468024 26843 tablet_replica.cc:330] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e: stopping tablet replica
I20260809 20:26:54.468300 26842 raft_consensus.cc:2229] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:26:54.468788 26843 raft_consensus.cc:2229] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:26:54.469466 26843 raft_consensus.cc:2258] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:26:54.470103 26844 ts_tablet_manager.cc:1905] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:26:54.472386 26843 ts_tablet_manager.cc:1905] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:26:54.472456 26842 raft_consensus.cc:2258] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:26:54.478694 26842 ts_tablet_manager.cc:1905] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:26:54.481230 26843 ts_tablet_manager.cc:1918] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:26:54.481492 26843 log.cc:1196] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/d716c80d604b4c07b30d5238aa7d5e12
I20260809 20:26:54.482169 26843 ts_tablet_manager.cc:1939] T d716c80d604b4c07b30d5238aa7d5e12 P 9f2957a10aed40d1a575fd510ca0a61e: Deleting consensus metadata
I20260809 20:26:54.483693 26844 ts_tablet_manager.cc:1918] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:26:54.483865 26844 log.cc:1196] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/d716c80d604b4c07b30d5238aa7d5e12
I20260809 20:26:54.484519 26844 ts_tablet_manager.cc:1939] T d716c80d604b4c07b30d5238aa7d5e12 P d42c4ce4de254a5382d23c5e01f7a513: Deleting consensus metadata
I20260809 20:26:54.484443 26262 catalog_manager.cc:4841] TS 9f2957a10aed40d1a575fd510ca0a61e (127.12.78.3:43465): tablet d716c80d604b4c07b30d5238aa7d5e12 (table test-table [id=c18f0b961dbc4ce7bf07128deb36fe08]) successfully deleted
I20260809 20:26:54.486518 26261 catalog_manager.cc:4841] TS d42c4ce4de254a5382d23c5e01f7a513 (127.12.78.1:35775): tablet d716c80d604b4c07b30d5238aa7d5e12 (table test-table [id=c18f0b961dbc4ce7bf07128deb36fe08]) successfully deleted
I20260809 20:26:54.488227 26816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.465980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54716 (local address 127.12.78.1:35775)
0809 20:26:54.466168 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:54.466175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:54.466282 (+ 107us) server_negotiation.cc:407] Connection header received
0809 20:26:54.466516 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:54.466527 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:54.466594 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:54.466711 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:54.468646 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.469435 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.471160 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.471570 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.471716 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.475424 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:54.475435 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:54.475449 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:54.475498 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:54.479511 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:54.485715 (+ 6204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:54.485724 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:54.485728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:54.485945 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:54.486437 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:54.486441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:54.486444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:54.486904 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:54.487253 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:54.487581 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 20:26:54.487830 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 20:26:54.488641 26842 ts_tablet_manager.cc:1918] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:26:54.488819 26842 log.cc:1196] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/d716c80d604b4c07b30d5238aa7d5e12
I20260809 20:26:54.488988 26484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:26:54.489387 26842 ts_tablet_manager.cc:1939] T d716c80d604b4c07b30d5238aa7d5e12 P c816db082dd64c86bad91cde1c3d46b0: Deleting consensus metadata
I20260809 20:26:54.491662 26260 catalog_manager.cc:4841] TS c816db082dd64c86bad91cde1c3d46b0 (127.12.78.2:35435): tablet d716c80d604b4c07b30d5238aa7d5e12 (table test-table [id=c18f0b961dbc4ce7bf07128deb36fe08]) successfully deleted
I20260809 20:26:54.581965 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 26317
I20260809 20:26:54.602272 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 26494
I20260809 20:26:54.619576 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.603335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.12.78.2:35435)
0809 20:26:54.603561 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:54.603567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:54.603593 (+ 26us) server_negotiation.cc:407] Connection header received
0809 20:26:54.603742 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:54.603749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:54.603841 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:54.603979 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:54.605381 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.606660 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.608138 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.609067 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.609215 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.611952 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:54.611974 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:54.611997 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:54.612055 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:54.616304 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:54.617197 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:54.617208 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:54.617213 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:54.617385 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:54.617798 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:54.617804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:54.617807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:54.618328 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:54.618675 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:54.618870 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 20:26:54.619163 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 20:26:54.620462 26637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:26:54.724928 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 26494
I20260809 20:26:54.753803 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 26647
I20260809 20:26:54.771898 26814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:54.754986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.12.78.3:43465)
0809 20:26:54.755245 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:54.755252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:54.755278 (+ 26us) server_negotiation.cc:407] Connection header received
0809 20:26:54.755352 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:54.755360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:54.755461 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:54.755596 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:54.757057 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.758448 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.759842 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:54.760672 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:54.760821 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:54.763667 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:54.763682 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:54.763703 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:54.763765 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:54.768481 (+ 4716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:54.769369 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:54.769380 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:54.769384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:54.769611 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:54.770011 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:54.770025 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:54.770030 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:54.770547 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:54.770940 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:54.771125 (+ 185us) server_negotiation.cc:299] Negotiation successful
0809 20:26:54.771405 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 20:26:54.772877 26782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:26:54.905903 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 26647
I20260809 20:26:54.930339 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 26241
I20260809 20:26:54.931532 26310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:26:55.045545 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 26241
I20260809 20:26:55.070286 12600 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 20:26:55.416599 12600 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 20:26:55.416828 12600 mini_ranger.cc:66] Stopping Ranger...
I20260809 20:26:55.762863 12600 mini_ranger.cc:68] Stopped Ranger
2026-08-09 20:26:55.763 UTC [25232] LOG: received smart shutdown request
2026-08-09 20:26:55.766 UTC [25232] LOG: background worker "logical replication launcher" (PID 25242) exited with exit code 1
2026-08-09 20:26:55.766 UTC [25236] LOG: shutting down
2026-08-09 20:26:55.941 UTC [25232] LOG: database system is shut down
2026-08-09T20:26:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55449 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:26:56 dist-test-slave-rnrw krb5kdc[26859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:26:56 dist-test-slave-rnrw krb5kdc[26859](info): set up 2 sockets
Aug 09 20:26:56 dist-test-slave-rnrw krb5kdc[26859](info): commencing operation
krb5kdc: starting...
W20260809 20:26:57.316206 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:26:57 dist-test-slave-rnrw krb5kdc[26859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:26:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:26:57.592185 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:41553
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:41553,127.12.78.61:41121,127.12.78.60:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:26:57.941342 26872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:26:57.941663 26872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:26:57.941761 26872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:26:57.949051 26872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:26:57.949169 26872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:26:57.949215 26872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:26:57.949273 26872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:26:57.949317 26872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:26:57.959271 26872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:41553,127.12.78.61:41121,127.12.78.60:46201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:41553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:26:57.960561 26872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:26:57.961797 26872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:26:57.973806 26877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:57.973981 26878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:57.974014 26880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:57.975083 26872 server_base.cc:1034] running on GCE node
I20260809 20:26:57.975889 26872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:26:57.977442 26872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:26:57.978682 26872 hybrid_clock.cc:648] HybridClock initialized: now 1786307217978633 us; error 65 us; skew 500 ppm
I20260809 20:26:57.979049 26872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:26:57 dist-test-slave-rnrw krb5kdc[26859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307217, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:26:57.982285 26872 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:26:57.983878 26872 webserver.cc:460] Webserver started at http://127.12.78.62:38823/ using document root <none> and password file <none>
I20260809 20:26:57.984694 26872 fs_manager.cc:359] Metadata directory not provided
I20260809 20:26:57.984844 26872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:26:57.985237 26872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:26:57.988312 26872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "c6737b50771e4f64b152cacfa41ff2f8"
format_stamp: "Formatted at 2026-08-09 20:26:57 on dist-test-slave-rnrw"
server_key: "71d189f5f11675c1fe79429e7b68228b"
server_key_iv: "f6d633042b85f69159a3156794aef58a"
server_key_version: "encryptionkey@0"
I20260809 20:26:57.989267 26872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "c6737b50771e4f64b152cacfa41ff2f8"
format_stamp: "Formatted at 2026-08-09 20:26:57 on dist-test-slave-rnrw"
server_key: "71d189f5f11675c1fe79429e7b68228b"
server_key_iv: "f6d633042b85f69159a3156794aef58a"
server_key_version: "encryptionkey@0"
I20260809 20:26:57.994930 26872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:26:57.998740 26887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:58.000196 26872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 20:26:58.000403 26872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "c6737b50771e4f64b152cacfa41ff2f8"
format_stamp: "Formatted at 2026-08-09 20:26:57 on dist-test-slave-rnrw"
server_key: "71d189f5f11675c1fe79429e7b68228b"
server_key_iv: "f6d633042b85f69159a3156794aef58a"
server_key_version: "encryptionkey@0"
I20260809 20:26:58.000687 26872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:26:58.016803 26872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:26:58.021152 26872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:26:58.021567 26872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:26:58.041421 26872 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:41553
I20260809 20:26:58.041458 26940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:41553 every 8 connection(s)
I20260809 20:26:58.042824 26872 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:26:58.046546 26942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:26:58.049541 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 26872
I20260809 20:26:58.049786 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:26:58.050639 12600 external_mini_cluster.cc:1428] Setting key 5bfba3dfdb3c5febd45368b4514208a1
I20260809 20:26:58.056519 26942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:26:58 dist-test-slave-rnrw krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307217, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.78.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.61@KRBTEST.COM" created.
I20260809 20:26:58.082538 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.060616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:41553 (local address 127.0.0.1:33412)
0809 20:26:58.061510 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.061570 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:58.063747 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:58.064636 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:58.064646 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:58.065458 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:58.066023 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:58.066057 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.067619 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.067627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:58.069633 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:58.069659 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.070637 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.070647 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:58.070864 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:58.071641 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:58.071675 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:58.074459 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:58.078041 (+ 3582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:58.078061 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:58.078073 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:58.078506 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:58.078985 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:58.078991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:58.078997 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:58.079158 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:58.080018 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:58.080030 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:58.080259 (+ 229us) client_negotiation.cc:769] Sending connection context
0809 20:26:58.080508 (+ 249us) client_negotiation.cc:240] Negotiation successful
0809 20:26:58.081151 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":153,"thread_start_us":408,"threads_started":1}
I20260809 20:26:58.082541 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.060621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.12.78.62:41553)
0809 20:26:58.061995 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:58.062005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:58.062060 (+ 55us) server_negotiation.cc:407] Connection header received
0809 20:26:58.064134 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:58.064139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:58.064369 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:58.064537 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:58.066238 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.067454 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.069798 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.070377 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.070508 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:58.074776 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:58.074789 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:58.074810 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:58.074856 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:58.077894 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:58.078650 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:58.078656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:58.078658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:58.078868 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:58.079271 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:58.079275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:58.079279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:58.079875 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:58.080100 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:58.080672 (+ 572us) server_negotiation.cc:299] Negotiation successful
0809 20:26:58.081026 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1157,"thread_start_us":952,"threads_started":1}
I20260809 20:26:58.085860 26942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } has no permanent_uuid. Determining permanent_uuid...
I20260809 20:26:58.087126 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.086721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:41121 (local address 127.0.0.1:45838)
0809 20:26:58.086894 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.087005 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260809 20:26:58.087410 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
I20260809 20:26:58.089100 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.088740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:41121 (local address 127.0.0.1:45850)
0809 20:26:58.088895 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.088961 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 20:26:58.089321 26942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:41121: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.61.keytab.
Entry for principal kudu/127.12.78.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.61.keytab.
I20260809 20:26:58.112703 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.61:41121
--webserver_interface=127.12.78.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.61.keytab
--principal=kudu/127.12.78.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:41553,127.12.78.61:41121,127.12.78.60:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:26:58.138726 26942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } attempt: 1
I20260809 20:26:58.139899 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.139428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:41121 (local address 127.0.0.1:45854)
0809 20:26:58.139663 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.139769 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 20:26:58.141408 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.140818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:41121 (local address 127.0.0.1:45870)
0809 20:26:58.141088 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.141196 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260809 20:26:58.141633 26942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:41121: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
I20260809 20:26:58.209885 26942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } attempt: 2
I20260809 20:26:58.211035 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.210575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:41121 (local address 127.0.0.1:45878)
0809 20:26:58.210784 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.210896 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260809 20:26:58.212150 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.211775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:41121 (local address 127.0.0.1:45886)
0809 20:26:58.211928 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.212011 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 20:26:58.212343 26942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:41121: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
I20260809 20:26:58.303797 26942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } attempt: 3
I20260809 20:26:58.305088 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.304625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:41121 (local address 127.0.0.1:45900)
0809 20:26:58.304819 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.304942 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 20:26:58.306224 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.305854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:41121 (local address 127.0.0.1:45902)
0809 20:26:58.305997 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.306085 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 20:26:58.306419 26942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:41121: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
I20260809 20:26:58.449680 26942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } attempt: 4
I20260809 20:26:58.450930 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.450351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:41121 (local address 127.0.0.1:35740)
0809 20:26:58.450715 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.450799 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
I20260809 20:26:58.452028 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.451660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:41121 (local address 127.0.0.1:35744)
0809 20:26:58.451812 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.451888 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 20:26:58.452224 26942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:41121: Network error: Client connection negotiation failed: client connection to 127.12.78.61:41121: connect: Connection refused (error 111)
W20260809 20:26:58.456789 26951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:26:58.457116 26951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:26:58.457289 26951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:26:58.464519 26951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:26:58.464700 26951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:26:58.464805 26951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:26:58.464916 26951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:26:58.465004 26951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:26:58.474995 26951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:41553,127.12.78.61:41121,127.12.78.60:46201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.61.keytab
--principal=kudu/127.12.78.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.61:41121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.78.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:26:58.476410 26951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:26:58.477798 26951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:26:58.489970 26959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:58.489970 26956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:58.490067 26957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:58.494202 26951 server_base.cc:1034] running on GCE node
I20260809 20:26:58.495255 26951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:26:58.496843 26951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:26:58.498070 26951 hybrid_clock.cc:648] HybridClock initialized: now 1786307218498032 us; error 52 us; skew 500 ppm
I20260809 20:26:58.498399 26951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:26:58 dist-test-slave-rnrw krb5kdc[26859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307218, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:26:58.501716 26951 init.cc:377] Logged in from keytab as kudu/127.12.78.61@KRBTEST.COM (short username kudu)
I20260809 20:26:58.503238 26951 webserver.cc:460] Webserver started at http://127.12.78.61:38681/ using document root <none> and password file <none>
I20260809 20:26:58.504000 26951 fs_manager.cc:359] Metadata directory not provided
I20260809 20:26:58.504112 26951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:26:58.504402 26951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:26:58.507714 26951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/data/instance:
uuid: "7f89991efb4b41edb75e99e7ab1b4523"
format_stamp: "Formatted at 2026-08-09 20:26:58 on dist-test-slave-rnrw"
server_key: "9fa0840483be64e0c009a49dc570f080"
server_key_iv: "4e661288ae92894773fc4adf036eb9e3"
server_key_version: "encryptionkey@0"
I20260809 20:26:58.508491 26951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/wal/instance:
uuid: "7f89991efb4b41edb75e99e7ab1b4523"
format_stamp: "Formatted at 2026-08-09 20:26:58 on dist-test-slave-rnrw"
server_key: "9fa0840483be64e0c009a49dc570f080"
server_key_iv: "4e661288ae92894773fc4adf036eb9e3"
server_key_version: "encryptionkey@0"
I20260809 20:26:58.513882 26951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 20:26:58.517305 26966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:58.518819 26951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:26:58.518980 26951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/wal
uuid: "7f89991efb4b41edb75e99e7ab1b4523"
format_stamp: "Formatted at 2026-08-09 20:26:58 on dist-test-slave-rnrw"
server_key: "9fa0840483be64e0c009a49dc570f080"
server_key_iv: "4e661288ae92894773fc4adf036eb9e3"
server_key_version: "encryptionkey@0"
I20260809 20:26:58.519171 26951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:26:58.560639 26951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:26:58.565514 26951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:26:58.565793 26951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:26:58.586853 26951 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.61:41121
I20260809 20:26:58.586861 27020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.61:41121 every 8 connection(s)
I20260809 20:26:58.588445 26951 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/data/info.pb
I20260809 20:26:58.589395 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 26951
I20260809 20:26:58.589625 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-1/wal/instance
I20260809 20:26:58.590185 12600 external_mini_cluster.cc:1428] Setting key b58aae2ea9944ecaea238eb7ef5adaaa
I20260809 20:26:58.593454 27021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:26:58.602977 27021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.78.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.60@KRBTEST.COM" created.
Aug 09 20:26:58 dist-test-slave-rnrw krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307218, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.61@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:26:58.626883 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.605891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42906 (local address 127.12.78.62:41553)
0809 20:26:58.606404 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:58.606409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:58.607524 (+ 1115us) server_negotiation.cc:407] Connection header received
0809 20:26:58.609577 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:58.609583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:58.609680 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:58.609805 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:58.611415 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.612145 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.613783 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.614204 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.614342 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:58.620218 (+ 5876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:58.620224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:58.620228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:58.620277 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:58.623349 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:58.624354 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:58.624358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:58.624361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:58.624509 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:58.625024 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:58.625031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:58.625039 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:58.625334 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:58.625509 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:58.626255 (+ 746us) server_negotiation.cc:299] Negotiation successful
0809 20:26:58.626513 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":293,"threads_started":1}
I20260809 20:26:58.627897 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.606475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:41553 (local address 127.0.0.1:42906)
0809 20:26:58.607306 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.607344 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:58.609284 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:58.610061 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:58.610069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:58.610626 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:58.611219 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:58.611237 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.612324 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.612329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:58.613631 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:58.613643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.614462 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.614471 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:58.614694 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:58.615493 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:58.615519 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:58.619927 (+ 4408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:58.623522 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:58.623533 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:58.623553 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:58.624203 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:58.624682 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:58.624688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:58.624692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:58.624888 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:58.625469 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:58.625478 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:58.625861 (+ 383us) client_negotiation.cc:769] Sending connection context
0809 20:26:58.626158 (+ 297us) client_negotiation.cc:240] Negotiation successful
0809 20:26:58.626686 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":382,"threads_started":1}
I20260809 20:26:58.629666 27021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:26:58 dist-test-slave-rnrw krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307218, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.61@KRBTEST.COM for kudu/127.12.78.61@KRBTEST.COM
I20260809 20:26:58.646606 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.630550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:41121 (local address 127.0.0.1:35748)
0809 20:26:58.630769 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.630795 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:58.630933 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:58.631786 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:58.631789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:58.632138 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:58.632271 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:58.632279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.634337 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.634341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:58.635190 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:58.635201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.636441 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.636446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:58.636666 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:58.637267 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:58.637284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:58.639363 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:58.643925 (+ 4562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:58.643928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:58.643931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:58.644307 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:58.644796 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:58.644800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:58.644802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:58.644952 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:58.645642 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:58.645650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:58.645794 (+ 144us) client_negotiation.cc:769] Sending connection context
0809 20:26:58.645919 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 20:26:58.646260 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 20:26:58.646936 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.630579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.12.78.61:41121)
0809 20:26:58.631287 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:58.631298 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:58.631333 (+ 35us) server_negotiation.cc:407] Connection header received
0809 20:26:58.631391 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:58.631402 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:58.631631 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:58.631783 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:58.632441 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.634200 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.635332 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.636297 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.636432 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:58.639603 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:58.639620 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:58.639638 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:58.639680 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:58.643724 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:58.644449 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:58.644460 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:58.644463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:58.644658 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:58.645062 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:58.645066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:58.645068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:58.645436 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:58.645702 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:58.646089 (+ 387us) server_negotiation.cc:299] Negotiation successful
0809 20:26:58.646384 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":504,"thread_start_us":287,"threads_started":1}
I20260809 20:26:58.649369 27021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } has no permanent_uuid. Determining permanent_uuid...
I20260809 20:26:58.650733 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.650195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:37930)
0809 20:26:58.650356 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.650496 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 20:26:58.651054 26969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
I20260809 20:26:58.652697 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.652310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:37936)
0809 20:26:58.652464 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.652586 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 20:26:58.652962 27021 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:46201: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.60.keytab.
Entry for principal kudu/127.12.78.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.60.keytab.
I20260809 20:26:58.654209 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.60:46201
--webserver_interface=127.12.78.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.60.keytab
--principal=kudu/127.12.78.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:41553,127.12.78.61:41121,127.12.78.60:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:26:58.702322 27021 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } attempt: 1
I20260809 20:26:58.703588 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.703102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:37942)
0809 20:26:58.703297 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.703435 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 20:26:58.704737 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.704331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:37944)
0809 20:26:58.704479 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.704631 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 20:26:58.705009 27021 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:46201: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
I20260809 20:26:58.751468 26942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } attempt: 5
Aug 09 20:26:58 dist-test-slave-rnrw krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307217, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for kudu/127.12.78.61@KRBTEST.COM
I20260809 20:26:58.764742 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.752336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.12.78.61:41121)
0809 20:26:58.752539 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:58.752547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:58.752616 (+ 69us) server_negotiation.cc:407] Connection header received
0809 20:26:58.752834 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:58.752838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:58.752901 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:58.753007 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:58.753826 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.754513 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.755701 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.756160 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.756294 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:58.759463 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:58.759468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:58.759471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:58.759503 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:58.762114 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:58.762776 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:58.762783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:58.762787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:58.762913 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:58.763291 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:58.763297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:58.763301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:58.763591 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:58.763747 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:58.764156 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 20:26:58.764372 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 20:26:58.764742 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.752266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:41121 (local address 127.0.0.1:35754)
0809 20:26:58.752490 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.752512 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:58.752685 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:58.753039 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:58.753044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:58.753461 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:58.753650 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:58.753659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.754662 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.754672 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:58.755550 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:58.755561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:58.756337 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:58.756343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:58.756523 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:58.757208 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:58.757226 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:58.759261 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:58.762259 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:58.762264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:58.762268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:58.762659 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:58.763031 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:58.763036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:58.763040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:58.763183 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:58.763710 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:58.763714 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:58.763870 (+ 156us) client_negotiation.cc:769] Sending connection context
0809 20:26:58.764038 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 20:26:58.764342 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260809 20:26:58.766117 26942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } has no permanent_uuid. Determining permanent_uuid...
I20260809 20:26:58.767320 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.766926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:37958)
0809 20:26:58.767097 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.767175 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 20:26:58.768337 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.767998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:37966)
0809 20:26:58.768158 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.768238 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 20:26:58.768604 26942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:46201: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
I20260809 20:26:58.773221 27021 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } attempt: 2
I20260809 20:26:58.774302 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.773855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:37980)
0809 20:26:58.774070 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.774160 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260809 20:26:58.775229 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.774905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:37988)
0809 20:26:58.775043 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.775115 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 20:26:58.775426 27021 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:46201: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
I20260809 20:26:58.819849 26942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } attempt: 1
I20260809 20:26:58.821087 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.820635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:38002)
0809 20:26:58.820822 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.820928 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 20:26:58.822088 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.821748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:38010)
0809 20:26:58.821896 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.821978 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 20:26:58.822307 26942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:46201: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
I20260809 20:26:58.866674 27021 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } attempt: 3
I20260809 20:26:58.868043 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.867471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:38026)
0809 20:26:58.867759 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.867858 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260809 20:26:58.869323 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.868819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:38038)
0809 20:26:58.869102 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.869206 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260809 20:26:58.869588 27021 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:46201: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
I20260809 20:26:58.890508 26942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } attempt: 2
I20260809 20:26:58.891690 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.891189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:38040)
0809 20:26:58.891433 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.891523 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 20:26:58.892758 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.892375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:38054)
0809 20:26:58.892529 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:58.892644 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 20:26:58.892982 26942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:46201: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
I20260809 20:26:58.999212 26942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } attempt: 3
I20260809 20:26:59.000461 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:58.999969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:38066)
0809 20:26:59.000182 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:59.000283 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 20:26:59.001582 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.001170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:38082)
0809 20:26:59.001387 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:59.001461 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260809 20:26:59.001794 26942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:46201: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
W20260809 20:26:59.003486 27029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:26:59.003831 27029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:26:59.003984 27029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:26:59.011570 27029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:26:59.011739 27029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:26:59.011847 27029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:26:59.011951 27029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:26:59.012035 27029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:26:59.012795 27021 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } attempt: 4
I20260809 20:26:59.013969 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.013471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:38094)
0809 20:26:59.013729 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:59.013820 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 20:26:59.014979 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.014644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:38100)
0809 20:26:59.014796 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:59.014879 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 20:26:59.015220 27021 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:46201: Network error: Client connection negotiation failed: client connection to 127.12.78.60:46201: connect: Connection refused (error 111)
I20260809 20:26:59.023077 27029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:41553,127.12.78.61:41121,127.12.78.60:46201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.60.keytab
--principal=kudu/127.12.78.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.60:46201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.78.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:26:59.024535 27029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:26:59.025940 27029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:26:59.037734 27035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:59.037817 27034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:26:59.037798 27037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:26:59.039011 27029 server_base.cc:1034] running on GCE node
I20260809 20:26:59.039811 27029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:26:59.041349 27029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:26:59.042595 27029 hybrid_clock.cc:648] HybridClock initialized: now 1786307219042539 us; error 60 us; skew 500 ppm
I20260809 20:26:59.042936 27029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:26:59 dist-test-slave-rnrw krb5kdc[26859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307219, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:26:59.046147 27029 init.cc:377] Logged in from keytab as kudu/127.12.78.60@KRBTEST.COM (short username kudu)
I20260809 20:26:59.047717 27029 webserver.cc:460] Webserver started at http://127.12.78.60:45549/ using document root <none> and password file <none>
I20260809 20:26:59.048396 27029 fs_manager.cc:359] Metadata directory not provided
I20260809 20:26:59.048530 27029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:26:59.048882 27029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:26:59.051877 27029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/data/instance:
uuid: "ac5aa24bd0a84fbe862952ac6230bf23"
format_stamp: "Formatted at 2026-08-09 20:26:59 on dist-test-slave-rnrw"
server_key: "f04f3c73bf36dd1328dbc0b22b9abf22"
server_key_iv: "7d8c1ea62ae0bdad6d5dcf150db1955c"
server_key_version: "encryptionkey@0"
I20260809 20:26:59.052758 27029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/wal/instance:
uuid: "ac5aa24bd0a84fbe862952ac6230bf23"
format_stamp: "Formatted at 2026-08-09 20:26:59 on dist-test-slave-rnrw"
server_key: "f04f3c73bf36dd1328dbc0b22b9abf22"
server_key_iv: "7d8c1ea62ae0bdad6d5dcf150db1955c"
server_key_version: "encryptionkey@0"
I20260809 20:26:59.057732 27029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 20:26:59.061152 27044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:26:59.062366 27029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:26:59.062542 27029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/wal
uuid: "ac5aa24bd0a84fbe862952ac6230bf23"
format_stamp: "Formatted at 2026-08-09 20:26:59 on dist-test-slave-rnrw"
server_key: "f04f3c73bf36dd1328dbc0b22b9abf22"
server_key_iv: "7d8c1ea62ae0bdad6d5dcf150db1955c"
server_key_version: "encryptionkey@0"
I20260809 20:26:59.062743 27029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:26:59.087070 27029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:26:59.091372 27029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:26:59.091703 27029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:26:59.110357 27029 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.60:46201
I20260809 20:26:59.110414 27096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.60:46201 every 8 connection(s)
I20260809 20:26:59.111773 27029 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/data/info.pb
I20260809 20:26:59.115506 27097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:26:59.121850 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 27029
I20260809 20:26:59.122071 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786307001768029-12600-0/minicluster-data/master-2/wal/instance
I20260809 20:26:59.122715 12600 external_mini_cluster.cc:1428] Setting key da651659951cf73902f1ea9801b09508
I20260809 20:26:59.122225 27097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:26:59 dist-test-slave-rnrw krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307219, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.60@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
Aug 09 20:26:59 dist-test-slave-rnrw krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.60@KRBTEST.COM
I20260809 20:26:59.144274 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.126167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42914 (local address 127.12.78.62:41553)
0809 20:26:59.126419 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:59.126427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:59.126948 (+ 521us) server_negotiation.cc:407] Connection header received
0809 20:26:59.128797 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:59.128801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:59.128881 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:59.128966 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:59.130379 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.131149 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.132988 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.133670 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.133807 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.137709 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:59.137715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:59.137718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:59.137753 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:59.141116 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.141929 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.141934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.141938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.142091 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.142524 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.142529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.142532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.142819 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:59.142978 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:59.143667 (+ 689us) server_negotiation.cc:299] Negotiation successful
0809 20:26:59.143909 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 20:26:59.145444 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.125142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.12.78.60:46201)
0809 20:26:59.125813 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:59.125824 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:59.126648 (+ 824us) server_negotiation.cc:407] Connection header received
0809 20:26:59.128502 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:59.128564 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:59.128865 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:59.129074 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:59.130562 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.132072 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.133786 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.134693 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.134782 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.139417 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:59.139424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:59.139436 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:59.139468 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:59.142247 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.142970 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.142976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.142979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.143109 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.143509 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.143518 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.143520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.143847 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:59.144175 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:59.144828 (+ 653us) server_negotiation.cc:299] Negotiation successful
0809 20:26:59.145047 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":475,"thread_start_us":291,"threads_started":1}
I20260809 20:26:59.145466 27101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.126083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:41553 (local address 127.0.0.1:42914)
0809 20:26:59.126768 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:59.126834 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:59.128502 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:59.129071 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:59.129077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:59.129798 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:59.130189 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:59.130206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.131307 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.131313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:59.132851 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:59.132862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.133895 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.133902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:59.134279 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.135062 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:59.135083 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:59.137467 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:59.141269 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:59.141282 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:59.141295 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:59.141796 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:59.142208 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:59.142213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:59.142217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:59.142403 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:59.142959 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:59.142966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:59.143305 (+ 339us) client_negotiation.cc:769] Sending connection context
0809 20:26:59.143575 (+ 270us) client_negotiation.cc:240] Negotiation successful
0809 20:26:59.144114 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":64,"thread_start_us":306,"threads_started":1}
I20260809 20:26:59.147871 27097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:26:59 dist-test-slave-rnrw krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307219, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.60@KRBTEST.COM for kudu/127.12.78.61@KRBTEST.COM
Aug 09 20:26:59 dist-test-slave-rnrw krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:26:59.162380 27101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.148772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:41121 (local address 127.0.0.1:35766)
0809 20:26:59.148986 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:59.149000 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:59.149135 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:59.149462 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:59.149466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:59.149957 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:59.150153 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:59.150163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.151135 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.151143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:59.152005 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:59.152024 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.152778 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.152783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:59.153002 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.153629 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:59.153651 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:59.156103 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:59.159658 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:59.159667 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:59.159671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:59.160226 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:59.160785 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:59.160791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:59.160794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:59.160982 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:59.161555 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:59.161561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:59.161737 (+ 176us) client_negotiation.cc:769] Sending connection context
0809 20:26:59.161889 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 20:26:59.162143 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260809 20:26:59.162389 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.148738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.12.78.61:41121)
0809 20:26:59.148921 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:59.148927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:59.149096 (+ 169us) server_negotiation.cc:407] Connection header received
0809 20:26:59.149264 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:59.149276 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:59.149340 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:59.149443 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:59.150316 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.150998 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.152153 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.152638 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.152769 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.156332 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:59.156341 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:59.156347 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:59.156392 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:59.159510 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.160391 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.160395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.160397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.160660 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.161119 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.161123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.161125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.161404 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:59.161536 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:59.161957 (+ 421us) server_negotiation.cc:299] Negotiation successful
0809 20:26:59.162121 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 20:26:59.163614 27097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } has no permanent_uuid. Determining permanent_uuid...
I20260809 20:26:59.165333 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.148784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42918 (local address 127.12.78.62:41553)
0809 20:26:59.149101 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:59.149106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:59.149125 (+ 19us) server_negotiation.cc:407] Connection header received
0809 20:26:59.149206 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:59.149211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:59.149291 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:59.149397 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:59.150752 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.151459 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.152728 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.153151 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.153307 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.157867 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:59.157873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:59.157876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:59.157909 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:59.161081 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.162752 (+ 1671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.162758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.162761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.162931 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.163397 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.163402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.163405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.163727 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:59.163995 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:59.164687 (+ 692us) server_negotiation.cc:299] Negotiation successful
0809 20:26:59.164944 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
Aug 09 20:26:59 dist-test-slave-rnrw krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307219, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.60@KRBTEST.COM for kudu/127.12.78.60@KRBTEST.COM
Aug 09 20:26:59 dist-test-slave-rnrw krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.61@KRBTEST.COM
I20260809 20:26:59.179032 26942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } attempt: 4
I20260809 20:26:59.179505 27101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.164469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:38106)
0809 20:26:59.164729 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:59.164751 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:59.164849 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:59.165212 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:59.165216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:59.165751 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:59.165967 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:59.165976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.167016 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.167022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:59.168335 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:59.168349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.169107 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.169112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:59.169305 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.169978 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:59.169995 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:59.172449 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:59.176670 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:59.176679 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:59.176686 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:59.177253 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:59.177706 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:59.177713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:59.177718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:59.177858 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:59.178472 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:59.178476 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:59.178613 (+ 137us) client_negotiation.cc:769] Sending connection context
0809 20:26:59.178743 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 20:26:59.179080 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260809 20:26:59.180693 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.164754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.12.78.60:46201)
0809 20:26:59.164916 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:59.164923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:59.164939 (+ 16us) server_negotiation.cc:407] Connection header received
0809 20:26:59.164985 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:59.164990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:59.165071 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:59.165198 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:59.166092 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.166871 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.168472 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.168956 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.169055 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.172718 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:59.172727 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:59.172732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:59.172770 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:59.176473 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.177409 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.177413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.177416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.177584 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.177965 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.177973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.177976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.178343 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:59.178508 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:59.178874 (+ 366us) server_negotiation.cc:299] Negotiation successful
0809 20:26:59.179295 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":57}
I20260809 20:26:59.185043 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.167458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.12.78.61:41121)
0809 20:26:59.167608 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:59.167612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:59.167626 (+ 14us) server_negotiation.cc:407] Connection header received
0809 20:26:59.167667 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:59.167670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:59.167721 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:59.167797 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:59.169374 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.170313 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.171889 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.172579 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.172702 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.178330 (+ 5628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:59.178337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:59.178341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:59.178388 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:59.182028 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.182836 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.182843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.182847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.183008 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.183423 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.183428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.183431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.183798 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:59.184092 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:59.184422 (+ 330us) server_negotiation.cc:299] Negotiation successful
0809 20:26:59.184682 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 20:26:59.187358 27097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23: Bootstrap starting.
Aug 09 20:26:59 dist-test-slave-rnrw krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307217, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for kudu/127.12.78.60@KRBTEST.COM
I20260809 20:26:59.190238 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:26:59.191783 27097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23: Neither blocks nor log segments found. Creating new log.
I20260809 20:26:59.193320 27097 log.cc:824] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23: Log is configured to *not* fsync() on all Append() calls
I20260809 20:26:59.199083 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.179696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:38118)
0809 20:26:59.180094 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:59.180117 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:59.180251 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:59.180990 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:59.180995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:59.181551 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:59.181781 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:59.181795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.183665 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.183672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:59.185097 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:59.185115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.186022 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.186028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:59.186235 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.187090 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:59.187114 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:59.190292 (+ 3178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:59.194776 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:59.194785 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:59.194793 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:59.195507 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:59.196142 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:59.196150 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:59.196157 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:59.196374 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:59.197683 (+ 1309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:59.197692 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:59.197924 (+ 232us) client_negotiation.cc:769] Sending connection context
0809 20:26:59.198086 (+ 162us) client_negotiation.cc:240] Negotiation successful
0809 20:26:59.198500 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260809 20:26:59.200496 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.179750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38118 (local address 127.12.78.60:46201)
0809 20:26:59.180535 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:59.180604 (+ 69us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:59.180634 (+ 30us) server_negotiation.cc:407] Connection header received
0809 20:26:59.180681 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:59.180684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:59.180781 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:59.181928 (+ 1147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:59.182193 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.183489 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.185327 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.185862 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.186006 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.190559 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:59.190566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:59.190571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:59.190611 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:59.194546 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.195745 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.195754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.195760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.195975 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.196615 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.196622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.196629 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.197492 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:59.199606 (+ 2114us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:59.199787 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 20:26:59.200067 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":421,"threads_started":1}
I20260809 20:26:59.202674 27097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23: No bootstrap required, opened a new log
I20260809 20:26:59.207916 26942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8: Bootstrap starting.
I20260809 20:26:59.212239 26942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8: Neither blocks nor log segments found. Creating new log.
I20260809 20:26:59.213775 26942 log.cc:824] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8: Log is configured to *not* fsync() on all Append() calls
I20260809 20:26:59.213740 27097 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } }
I20260809 20:26:59.214191 27097 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:26:59.214319 27097 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5aa24bd0a84fbe862952ac6230bf23, State: Initialized, Role: FOLLOWER
I20260809 20:26:59.215010 27097 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } }
I20260809 20:26:59.219689 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.196699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.12.78.62:41553)
0809 20:26:59.196875 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:59.196881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:59.196900 (+ 19us) server_negotiation.cc:407] Connection header received
0809 20:26:59.196953 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:59.196959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:59.197038 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:59.197136 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:59.200342 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.201772 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.204566 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.205638 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.205759 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.211741 (+ 5982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:59.211750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:59.211757 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:59.211804 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:59.215666 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.216953 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.216959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.216963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.217122 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.217642 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.217648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.217652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.218004 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:59.218322 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:59.218960 (+ 638us) server_negotiation.cc:299] Negotiation successful
0809 20:26:59.219237 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 20:26:59.223107 27112 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } } }
I20260809 20:26:59.223690 27112 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:26:59.224709 27097 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:26:59.225654 26942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8: No bootstrap required, opened a new log
I20260809 20:26:59.228798 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.197246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.12.78.60:46201)
0809 20:26:59.197461 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:59.197469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:59.210558 (+ 13089us) server_negotiation.cc:407] Connection header received
0809 20:26:59.211133 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:59.211142 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:59.211257 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:59.211338 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:59.213435 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.214647 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.217494 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.218231 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.218328 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.222105 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:59.222111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:59.222116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:59.222158 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:59.225907 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.226593 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.226599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.226603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.226754 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.227153 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.227158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.227161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.227472 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:59.227730 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:59.228048 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 20:26:59.228274 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 20:26:59.240296 26942 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } }
I20260809 20:26:59.240890 26942 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:26:59.241065 26942 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6737b50771e4f64b152cacfa41ff2f8, State: Initialized, Role: FOLLOWER
I20260809 20:26:59.241948 26942 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } }
I20260809 20:26:59.244666 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.203572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.12.78.61:41121)
0809 20:26:59.203748 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:59.203753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:59.214849 (+ 11096us) server_negotiation.cc:407] Connection header received
0809 20:26:59.215165 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:59.215173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:59.215259 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:59.215337 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:59.220249 (+ 4912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.221741 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.233248 (+ 11507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.234159 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.234319 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.237531 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:59.237537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:59.237540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:59.237578 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:59.240176 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.241418 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.241424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.241427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.241622 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.242176 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.242183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.242185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.242611 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:59.242930 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:59.243868 (+ 938us) server_negotiation.cc:299] Negotiation successful
0809 20:26:59.244136 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 20:26:59.249331 27124 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } } }
I20260809 20:26:59.249790 27124 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:26:59.251860 26942 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:26:59.245265 27029 master_runner.cc:186] Error getting master registration for 127.12.78.62:41553: OK, instance_id { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" instance_seqno: 1786307218035925 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 20:26:59.257809 27029 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 20:26:59.258451 27123 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 20:26:59.258646 27123 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260809 20:26:59.274969 27137 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 20:26:59.275270 27137 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260809 20:26:59.275851 26893 tablet.cc:2369] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:26:59.272428 26872 master_runner.cc:186] Error getting master registration for 127.12.78.61:41121: OK, instance_id { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" instance_seqno: 1786307218580495 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 20:26:59.282444 26872 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 20:26:59.314424 27021 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } attempt: 5
Aug 09 20:26:59 dist-test-slave-rnrw krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307218, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.61@KRBTEST.COM for kudu/127.12.78.60@KRBTEST.COM
I20260809 20:26:59.331077 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.315149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.12.78.60:46201)
0809 20:26:59.315326 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 20:26:59.315331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:26:59.315445 (+ 114us) server_negotiation.cc:407] Connection header received
0809 20:26:59.315684 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:26:59.315688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:26:59.315770 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:26:59.315879 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:26:59.316742 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.317526 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.318813 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.319267 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.319407 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.324685 (+ 5278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:26:59.324694 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:26:59.324700 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:26:59.324736 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:26:59.328198 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.329020 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.329027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.329031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.329189 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:26:59.329609 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:26:59.329613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:26:59.329615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:26:59.329896 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:26:59.330057 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 20:26:59.330433 (+ 376us) server_negotiation.cc:299] Negotiation successful
0809 20:26:59.330676 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 20:26:59.331073 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:26:59.315076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:46201 (local address 127.0.0.1:38136)
0809 20:26:59.315335 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0809 20:26:59.315357 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:26:59.315521 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:26:59.315915 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:26:59.315919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:26:59.316358 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:26:59.316522 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:59.316583 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.317665 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.317672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:59.318661 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:26:59.318672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:26:59.319417 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:26:59.319422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:26:59.319638 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:26:59.320270 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:26:59.320288 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:26:59.324433 (+ 4145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:26:59.328386 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:59.328392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:59.328396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:59.328886 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:59.329313 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:26:59.329319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:26:59.329322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:26:59.329476 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:26:59.330020 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:26:59.330025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:26:59.330183 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 20:26:59.330329 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 20:26:59.330639 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260809 20:26:59.336359 27021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523: Bootstrap starting.
I20260809 20:26:59.341315 27021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523: Neither blocks nor log segments found. Creating new log.
I20260809 20:26:59.342499 27021 log.cc:824] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523: Log is configured to *not* fsync() on all Append() calls
W20260809 20:26:59.343933 27050 tablet.cc:2369] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:26:59.346112 27021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523: No bootstrap required, opened a new log
I20260809 20:26:59.354072 27021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } }
I20260809 20:26:59.354444 27021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:26:59.354571 27021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f89991efb4b41edb75e99e7ab1b4523, State: Initialized, Role: FOLLOWER
I20260809 20:26:59.355283 27021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } }
I20260809 20:26:59.361764 27140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } } }
I20260809 20:26:59.362255 27140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:26:59.363067 27021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:26:59.384799 26951 master_runner.cc:263] No leader master found from master 7f89991efb4b41edb75e99e7ab1b4523
I20260809 20:26:59.384984 26951 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 20:26:59.390661 27152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 20:26:59.390816 27152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 20:26:59.500712 27124 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:26:59.501130 27124 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } }
I20260809 20:26:59.502991 27124 leader_election.cc:288] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f89991efb4b41edb75e99e7ab1b4523 (127.12.78.61:41121), ac5aa24bd0a84fbe862952ac6230bf23 (127.12.78.60:46201)
I20260809 20:26:59.503785 27071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6737b50771e4f64b152cacfa41ff2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" is_pre_election: true
I20260809 20:26:59.504499 27071 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6737b50771e4f64b152cacfa41ff2f8 in term 0.
I20260809 20:26:59.505290 26890 leader_election.cc:302] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac5aa24bd0a84fbe862952ac6230bf23, c6737b50771e4f64b152cacfa41ff2f8; no voters:
I20260809 20:26:59.505740 27124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:26:59.505916 27124 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:26:59.506042 27124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:26:59.509269 26994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6737b50771e4f64b152cacfa41ff2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f89991efb4b41edb75e99e7ab1b4523" is_pre_election: true
I20260809 20:26:59.509886 26994 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6737b50771e4f64b152cacfa41ff2f8 in term 0.
I20260809 20:26:59.512432 27124 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } }
I20260809 20:26:59.513619 26994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6737b50771e4f64b152cacfa41ff2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f89991efb4b41edb75e99e7ab1b4523"
I20260809 20:26:59.513841 26994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:26:59.514003 27124 leader_election.cc:288] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [CANDIDATE]: Term 1 election: Requested vote from peers 7f89991efb4b41edb75e99e7ab1b4523 (127.12.78.61:41121), ac5aa24bd0a84fbe862952ac6230bf23 (127.12.78.60:46201)
I20260809 20:26:59.515048 27071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6737b50771e4f64b152cacfa41ff2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5aa24bd0a84fbe862952ac6230bf23"
I20260809 20:26:59.515303 27071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:26:59.517356 26994 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6737b50771e4f64b152cacfa41ff2f8 in term 1.
I20260809 20:26:59.517855 26891 leader_election.cc:302] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f89991efb4b41edb75e99e7ab1b4523, c6737b50771e4f64b152cacfa41ff2f8; no voters:
I20260809 20:26:59.518247 27124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:26:59.519124 27071 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6737b50771e4f64b152cacfa41ff2f8 in term 1.
I20260809 20:26:59.520315 27124 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [term 1 LEADER]: Becoming Leader. State: Replica: c6737b50771e4f64b152cacfa41ff2f8, State: Running, Role: LEADER
I20260809 20:26:59.521005 27124 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } }
I20260809 20:26:59.533571 27154 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6737b50771e4f64b152cacfa41ff2f8. Latest consensus state: current_term: 1 leader_uuid: "c6737b50771e4f64b152cacfa41ff2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } } }
I20260809 20:26:59.534080 27154 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [sys.catalog]: This master's current role is: LEADER
I20260809 20:26:59.548653 27156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:26:59.551226 27156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260809 20:26:59.568595 26972 tablet.cc:2369] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:26:59.571838 27071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [term 1 FOLLOWER]: Refusing update from remote peer c6737b50771e4f64b152cacfa41ff2f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:26:59.572898 27154 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:26:59.580129 26994 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [term 1 FOLLOWER]: Refusing update from remote peer c6737b50771e4f64b152cacfa41ff2f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:26:59.581074 27154 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:26:59.618517 27140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6737b50771e4f64b152cacfa41ff2f8. Latest consensus state: current_term: 1 leader_uuid: "c6737b50771e4f64b152cacfa41ff2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } } }
I20260809 20:26:59.618852 27140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:26:59.620362 27157 mvcc.cc:205] Tried to move back new op lower bound from 7316714371355361280 to 7316714371163197440. Current Snapshot: MvccSnapshot[applied={T|T < 7316714371355361280}]
I20260809 20:26:59.625101 27159 mvcc.cc:205] Tried to move back new op lower bound from 7316714371355361280 to 7316714371163197440. Current Snapshot: MvccSnapshot[applied={T|T < 7316714371355361280}]
I20260809 20:26:59.632592 27124 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6737b50771e4f64b152cacfa41ff2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } } }
I20260809 20:26:59.632952 27124 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [sys.catalog]: This master's current role is: LEADER
I20260809 20:26:59.636062 27156 catalog_manager.cc:1347] Generated new cluster ID: 73a85347be5a4bfa9f8e9e36e33aecee
I20260809 20:26:59.636180 27156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:26:59.636395 27140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6737b50771e4f64b152cacfa41ff2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } } }
I20260809 20:26:59.636725 27140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:26:59.650648 27112 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6737b50771e4f64b152cacfa41ff2f8. Latest consensus state: current_term: 1 leader_uuid: "c6737b50771e4f64b152cacfa41ff2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } } }
I20260809 20:26:59.651036 27112 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:26:59.653939 27154 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6737b50771e4f64b152cacfa41ff2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } } }
I20260809 20:26:59.654249 27154 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8 [sys.catalog]: This master's current role is: LEADER
I20260809 20:26:59.656461 27158 mvcc.cc:205] Tried to move back new op lower bound from 7316714371355361280 to 7316714371163197440. Current Snapshot: MvccSnapshot[applied={T|T < 7316714371355361280}]
I20260809 20:26:59.669140 27112 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6737b50771e4f64b152cacfa41ff2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6737b50771e4f64b152cacfa41ff2f8" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 41553 } } peers { permanent_uuid: "7f89991efb4b41edb75e99e7ab1b4523" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 41121 } } peers { permanent_uuid: "ac5aa24bd0a84fbe862952ac6230bf23" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 46201 } } }
I20260809 20:26:59.669450 27112 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:26:59.682343 27156 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:26:59.684264 27156 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:26:59.726269 27156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6737b50771e4f64b152cacfa41ff2f8: Generated new TSK 0
I20260809 20:26:59.727393 27156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:27:00.260746 27123 catalog_manager.cc:1259] Loaded cluster ID: 73a85347be5a4bfa9f8e9e36e33aecee
I20260809 20:27:00.260859 27123 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23: loading cluster ID for follower catalog manager: success
I20260809 20:27:00.263381 27123 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23: acquiring CA information for follower catalog manager: success
I20260809 20:27:00.265492 27123 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac5aa24bd0a84fbe862952ac6230bf23: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 20:27:00.392975 27152 catalog_manager.cc:1259] Loaded cluster ID: 73a85347be5a4bfa9f8e9e36e33aecee
I20260809 20:27:00.393085 27152 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523: loading cluster ID for follower catalog manager: success
I20260809 20:27:00.395454 27152 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523: acquiring CA information for follower catalog manager: success
I20260809 20:27:00.397261 27152 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f89991efb4b41edb75e99e7ab1b4523: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 20:27:00.848944 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 26872
I20260809 20:27:00.850863 26936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:00.964386 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 26872
I20260809 20:27:00.988517 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 26951
I20260809 20:27:00.989876 27016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:01.098927 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 26951
I20260809 20:27:01.121876 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 27029
I20260809 20:27:01.123085 27092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:01.231748 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 27029
2026-08-09T20:27:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5162 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:27:01 dist-test-slave-rnrw krb5kdc[27176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:27:01 dist-test-slave-rnrw krb5kdc[27176](info): set up 2 sockets
Aug 09 20:27:01 dist-test-slave-rnrw krb5kdc[27176](info): commencing operation
krb5kdc: starting...
W20260809 20:27:02.494740 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.007s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:27:02 dist-test-slave-rnrw krb5kdc[27176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:27:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:27:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:27:02.772805 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44069
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:44069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:27:03.153908 27189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:03.154217 27189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:03.154309 27189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:03.161914 27189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:27:03.162029 27189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:03.162084 27189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:03.162143 27189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:27:03.162187 27189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:27:03.172333 27189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:44069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:44069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:03.173720 27189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:03.175045 27189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:03.187098 27196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:03.187288 27195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:03.187261 27198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:03.188226 27189 server_base.cc:1034] running on GCE node
I20260809 20:27:03.189030 27189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:03.190455 27189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:03.191673 27189 hybrid_clock.cc:648] HybridClock initialized: now 1786307223191606 us; error 49 us; skew 500 ppm
I20260809 20:27:03.192057 27189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:03 dist-test-slave-rnrw krb5kdc[27176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307223, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:03.195695 27189 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:27:03.197403 27189 webserver.cc:460] Webserver started at http://127.12.78.62:40923/ using document root <none> and password file <none>
I20260809 20:27:03.198079 27189 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:03.198184 27189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:03.198467 27189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:27:03.201589 27189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "ca2ad0264623485d8665f31bd9530737"
format_stamp: "Formatted at 2026-08-09 20:27:03 on dist-test-slave-rnrw"
server_key: "eed5be1bc73e51d9a0a0993a7b51472d"
server_key_iv: "8e11d599c663b9db20ccf9ec85349b10"
server_key_version: "encryptionkey@0"
I20260809 20:27:03.202323 27189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "ca2ad0264623485d8665f31bd9530737"
format_stamp: "Formatted at 2026-08-09 20:27:03 on dist-test-slave-rnrw"
server_key: "eed5be1bc73e51d9a0a0993a7b51472d"
server_key_iv: "8e11d599c663b9db20ccf9ec85349b10"
server_key_version: "encryptionkey@0"
I20260809 20:27:03.207569 27189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:27:03.211045 27205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:03.212441 27189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 20:27:03.212677 27189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "ca2ad0264623485d8665f31bd9530737"
format_stamp: "Formatted at 2026-08-09 20:27:03 on dist-test-slave-rnrw"
server_key: "eed5be1bc73e51d9a0a0993a7b51472d"
server_key_iv: "8e11d599c663b9db20ccf9ec85349b10"
server_key_version: "encryptionkey@0"
I20260809 20:27:03.212890 27189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:03.229473 27189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:03.234126 27189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:03.234467 27189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:03.256021 27189 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:44069
I20260809 20:27:03.256071 27259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:44069 every 8 connection(s)
I20260809 20:27:03.257593 27189 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:27:03.259624 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 27189
I20260809 20:27:03.259881 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:27:03.260612 12600 external_mini_cluster.cc:1428] Setting key c4ff9431ed147bf38a8ab310517b6d07
I20260809 20:27:03.262591 27260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 20:27:03 dist-test-slave-rnrw krb5kdc[27176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:03.278295 27260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737: Bootstrap starting.
I20260809 20:27:03.282230 27260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:03.283388 27260 log.cc:824] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:03.287108 27260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737: No bootstrap required, opened a new log
I20260809 20:27:03.287518 27263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:03.262677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.12.78.62:44069)
0809 20:27:03.263592 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:03.263609 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:03.263668 (+ 59us) server_negotiation.cc:407] Connection header received
0809 20:27:03.266130 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:03.266162 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:03.266739 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:03.267250 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:03.268748 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:03.271120 (+ 2372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:03.272623 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:03.273814 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:03.274046 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:03.278764 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:03.278772 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:03.278788 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:03.278833 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:03.282429 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:03.283319 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:03.283337 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:03.283344 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:03.283540 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:03.284121 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:03.284128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:03.284133 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:03.285119 (+ 986us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:03.285351 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:03.285931 (+ 580us) server_negotiation.cc:299] Negotiation successful
0809 20:27:03.286318 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":372,"threads_started":1}
I20260809 20:27:03.291956 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:27:03.296629 27260 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2ad0264623485d8665f31bd9530737" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44069 } }
I20260809 20:27:03.297089 27260 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:03.297215 27260 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2ad0264623485d8665f31bd9530737, State: Initialized, Role: FOLLOWER
I20260809 20:27:03.297894 27260 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2ad0264623485d8665f31bd9530737" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44069 } }
I20260809 20:27:03.298169 27260 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:27:03.298341 27260 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:27:03.298544 27260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:03.301103 27260 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2ad0264623485d8665f31bd9530737" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44069 } }
I20260809 20:27:03.301656 27260 leader_election.cc:302] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca2ad0264623485d8665f31bd9530737; no voters:
I20260809 20:27:03.302309 27260 leader_election.cc:288] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:27:03.302474 27265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:27:03.303913 27265 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [term 1 LEADER]: Becoming Leader. State: Replica: ca2ad0264623485d8665f31bd9530737, State: Running, Role: LEADER
I20260809 20:27:03.304486 27265 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2ad0264623485d8665f31bd9530737" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44069 } }
I20260809 20:27:03.304910 27260 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:27:03.307741 27266 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca2ad0264623485d8665f31bd9530737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2ad0264623485d8665f31bd9530737" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44069 } } }
I20260809 20:27:03.307737 27267 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca2ad0264623485d8665f31bd9530737. Latest consensus state: current_term: 1 leader_uuid: "ca2ad0264623485d8665f31bd9530737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2ad0264623485d8665f31bd9530737" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 44069 } } }
I20260809 20:27:03.308218 27266 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [sys.catalog]: This master's current role is: LEADER
I20260809 20:27:03.308244 27267 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737 [sys.catalog]: This master's current role is: LEADER
I20260809 20:27:03.310432 27272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:27:03.315080 27272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:27:03.324680 27272 catalog_manager.cc:1347] Generated new cluster ID: 9e211ba1980b473e94bb6533eaa197ad
I20260809 20:27:03.324820 27272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:27:03.337153 27272 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:27:03.338599 27272 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:27:03.345391 27272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca2ad0264623485d8665f31bd9530737: Generated new TSK 0
I20260809 20:27:03.346520 27272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:27:03.405140 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 27189
I20260809 20:27:03.406424 27255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:03.505860 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 27189
2026-08-09T20:27:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2270 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:27:03 dist-test-slave-rnrw krb5kdc[27289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:27:03 dist-test-slave-rnrw krb5kdc[27289](info): set up 2 sockets
Aug 09 20:27:03 dist-test-slave-rnrw krb5kdc[27289](info): commencing operation
krb5kdc: starting...
W20260809 20:27:04.664117 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:27:04 dist-test-slave-rnrw krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:27:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:27:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:27:04.956753 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40439
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:40439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:27:05.306463 27303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:05.306833 27303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:05.307008 27303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:05.314328 27303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:27:05.314487 27303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:05.314605 27303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:05.314715 27303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:27:05.314785 27303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:27:05.324698 27303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:40439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:05.326129 27303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:05.327513 27303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:05.339475 27308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:05.339550 27311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:05.339576 27309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:05.340647 27303 server_base.cc:1034] running on GCE node
I20260809 20:27:05.341487 27303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:05.342980 27303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:05.344228 27303 hybrid_clock.cc:648] HybridClock initialized: now 1786307225344172 us; error 40 us; skew 500 ppm
I20260809 20:27:05.344672 27303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:05 dist-test-slave-rnrw krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307225, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:05.348079 27303 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:27:05.349607 27303 webserver.cc:460] Webserver started at http://127.12.78.62:44707/ using document root <none> and password file <none>
I20260809 20:27:05.350301 27303 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:05.350406 27303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:05.350731 27303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:27:05.353902 27303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "0fff14088a4e4dcdae437f3723410567"
format_stamp: "Formatted at 2026-08-09 20:27:05 on dist-test-slave-rnrw"
server_key: "ce18f090a1f299fb4aea233f59cef1ad"
server_key_iv: "bbd8c6dc22f6152b71638c6a7670b5ae"
server_key_version: "encryptionkey@0"
I20260809 20:27:05.354699 27303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "0fff14088a4e4dcdae437f3723410567"
format_stamp: "Formatted at 2026-08-09 20:27:05 on dist-test-slave-rnrw"
server_key: "ce18f090a1f299fb4aea233f59cef1ad"
server_key_iv: "bbd8c6dc22f6152b71638c6a7670b5ae"
server_key_version: "encryptionkey@0"
I20260809 20:27:05.359875 27303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:27:05.363399 27320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:05.365175 27303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 20:27:05.365386 27303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "0fff14088a4e4dcdae437f3723410567"
format_stamp: "Formatted at 2026-08-09 20:27:05 on dist-test-slave-rnrw"
server_key: "ce18f090a1f299fb4aea233f59cef1ad"
server_key_iv: "bbd8c6dc22f6152b71638c6a7670b5ae"
server_key_version: "encryptionkey@0"
I20260809 20:27:05.365672 27303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:05.393754 27303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:05.398739 27303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:05.399072 27303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:05.420111 27303 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:40439
I20260809 20:27:05.420212 27373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:40439 every 8 connection(s)
I20260809 20:27:05.421752 27303 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:27:05.425755 27374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:05.426129 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 27303
I20260809 20:27:05.426357 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:27:05.427073 12600 external_mini_cluster.cc:1428] Setting key e432daba8bd8b3d160c0091573e4db87
I20260809 20:27:05.441044 27374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Bootstrap starting.
Aug 09 20:27:05 dist-test-slave-rnrw krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:05.444240 27374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:05.445503 27374 log.cc:824] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:05.449198 27374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: No bootstrap required, opened a new log
I20260809 20:27:05.455181 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:05.429433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.12.78.62:40439)
0809 20:27:05.430348 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:05.430365 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:05.430434 (+ 69us) server_negotiation.cc:407] Connection header received
0809 20:27:05.432781 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:05.432823 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:05.433431 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:05.433943 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:05.435526 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:05.437721 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:05.439202 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:05.440290 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:05.440499 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:05.445411 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:05.445422 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:05.445442 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:05.445516 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:05.450352 (+ 4836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:05.451136 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:05.451147 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:05.451160 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:05.451375 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:05.451822 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:05.451830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:05.451837 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:05.452620 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:05.452942 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:05.453368 (+ 426us) server_negotiation.cc:299] Negotiation successful
0809 20:27:05.453839 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":400,"threads_started":1}
I20260809 20:27:05.459295 27374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } }
I20260809 20:27:05.459636 27374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:05.459702 27374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fff14088a4e4dcdae437f3723410567, State: Initialized, Role: FOLLOWER
I20260809 20:27:05.460250 27374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } }
I20260809 20:27:05.460454 27374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:27:05.460673 27374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:27:05.460850 27374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:05.463241 27374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } }
I20260809 20:27:05.463694 27374 leader_election.cc:302] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fff14088a4e4dcdae437f3723410567; no voters:
I20260809 20:27:05.464267 27374 leader_election.cc:288] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:27:05.464488 27379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:27:05.465276 27379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 1 LEADER]: Becoming Leader. State: Replica: 0fff14088a4e4dcdae437f3723410567, State: Running, Role: LEADER
I20260809 20:27:05.465829 27379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } }
I20260809 20:27:05.466455 27374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:27:05.468621 27381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fff14088a4e4dcdae437f3723410567. Latest consensus state: current_term: 1 leader_uuid: "0fff14088a4e4dcdae437f3723410567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } } }
I20260809 20:27:05.469174 27381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: This master's current role is: LEADER
I20260809 20:27:05.469009 27380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fff14088a4e4dcdae437f3723410567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } } }
I20260809 20:27:05.469368 27380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: This master's current role is: LEADER
I20260809 20:27:05.472388 27385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:27:05.476871 27385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:27:05.486294 27385 catalog_manager.cc:1347] Generated new cluster ID: 08dc62be39534972a5ce404cd5d9f274
I20260809 20:27:05.486420 27385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:27:05.497145 27385 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:27:05.499042 27385 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:27:05.506368 27385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Generated new TSK 0
I20260809 20:27:05.507406 27385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:27:05.580307 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40439
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:27:05.937741 27401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:05.938066 27401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:05.938164 27401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:05.946125 27401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:05.946269 27401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:05.946439 27401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:27:05.960014 27401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:05.961638 27401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:05.963124 27401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:05.978526 27406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:05.978500 27407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:05.978463 27409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:05.980259 27401 server_base.cc:1034] running on GCE node
I20260809 20:27:05.981112 27401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:05.982298 27401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:05.983616 27401 hybrid_clock.cc:648] HybridClock initialized: now 1786307225983545 us; error 62 us; skew 500 ppm
I20260809 20:27:05.984043 27401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:05 dist-test-slave-rnrw krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307225, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:05.987934 27401 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:27:05.989818 27401 webserver.cc:460] Webserver started at http://127.12.78.1:45049/ using document root <none> and password file <none>
I20260809 20:27:05.990634 27401 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:05.990803 27401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:05.991153 27401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:27:05.994513 27401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "8370621deb3d465d840273954440c8ec"
format_stamp: "Formatted at 2026-08-09 20:27:05 on dist-test-slave-rnrw"
server_key: "70b9dbf59576ea10709938359335128d"
server_key_iv: "b56599bbd1a9c7c1b5940d491e1f316e"
server_key_version: "encryptionkey@0"
I20260809 20:27:05.995430 27401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "8370621deb3d465d840273954440c8ec"
format_stamp: "Formatted at 2026-08-09 20:27:05 on dist-test-slave-rnrw"
server_key: "70b9dbf59576ea10709938359335128d"
server_key_iv: "b56599bbd1a9c7c1b5940d491e1f316e"
server_key_version: "encryptionkey@0"
I20260809 20:27:06.001626 27401 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 20:27:06.006294 27416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:06.008144 27401 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 20:27:06.008396 27401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "8370621deb3d465d840273954440c8ec"
format_stamp: "Formatted at 2026-08-09 20:27:05 on dist-test-slave-rnrw"
server_key: "70b9dbf59576ea10709938359335128d"
server_key_iv: "b56599bbd1a9c7c1b5940d491e1f316e"
server_key_version: "encryptionkey@0"
I20260809 20:27:06.008775 27401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:06.040795 27401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:06.045662 27401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:06.045953 27401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:06.047017 27401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:06.048686 27401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:06.048835 27401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:06.049019 27401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:06.049139 27401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 20:27:06.088429 27401 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:46619
I20260809 20:27:06.088477 27536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:46619 every 8 connection(s)
I20260809 20:27:06.089946 27401 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:27:06.100273 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 27401
I20260809 20:27:06.100653 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:27:06.101526 12600 external_mini_cluster.cc:1428] Setting key 5a93f1dfbf5cc03a5ab3121fb91f38a7
Aug 09 20:27:06 dist-test-slave-rnrw krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307225, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:06.114926 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:06.092977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:44463 (local address 127.12.78.62:40439)
0809 20:27:06.093476 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:06.093484 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:06.094636 (+ 1152us) server_negotiation.cc:407] Connection header received
0809 20:27:06.096744 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:06.096749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:06.096840 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:06.096944 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:06.098653 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:06.099466 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:06.101319 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:06.102057 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:06.102224 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:06.106850 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:06.106861 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:06.106869 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:06.106921 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:06.110871 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:06.111791 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:06.111799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:06.111805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:06.111977 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:06.112494 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:06.112502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:06.112508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:06.113009 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:06.113185 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:06.114095 (+ 910us) server_negotiation.cc:299] Negotiation successful
0809 20:27:06.114382 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":234,"threads_started":1}
I20260809 20:27:06.116336 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:06.093576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:44463)
0809 20:27:06.094446 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:06.094496 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:06.096375 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:06.097233 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:06.097244 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:06.097832 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:27:06.098438 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:06.098455 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:06.099626 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:06.099630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:06.101121 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:06.101140 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:06.102518 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:06.102530 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:06.102861 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:06.103694 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:27:06.103720 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:27:06.106535 (+ 2815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:27:06.111068 (+ 4533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:06.111078 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:06.111095 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:06.111627 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:06.112134 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:06.112138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:06.112140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:06.112362 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:06.113203 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:27:06.113214 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:27:06.113630 (+ 416us) client_negotiation.cc:769] Sending connection context
0809 20:27:06.113922 (+ 292us) client_negotiation.cc:240] Negotiation successful
0809 20:27:06.114585 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":374,"threads_started":1}
I20260809 20:27:06.118268 27537 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40439
I20260809 20:27:06.118634 27537 heartbeater.cc:464] Registering TS with master...
I20260809 20:27:06.119635 27537 heartbeater.cc:511] Master 127.12.78.62:40439 requested a full tablet report, sending...
I20260809 20:27:06.122203 27338 ts_manager.cc:194] Registered new tserver with Master: 8370621deb3d465d840273954440c8ec (127.12.78.1:46619)
I20260809 20:27:06.123770 27338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:44463
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:27:06.169793 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40439
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:27:06.519511 27543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:06.519871 27543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:06.520040 27543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:06.527477 27543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:06.527638 27543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:06.527812 27543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:27:06.537582 27543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:06.538897 27543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:06.540225 27543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:06.553184 27548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:06.553259 27551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:06.553280 27549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:06.554131 27543 server_base.cc:1034] running on GCE node
I20260809 20:27:06.554718 27543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:06.555635 27543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:06.556870 27543 hybrid_clock.cc:648] HybridClock initialized: now 1786307226556824 us; error 49 us; skew 500 ppm
I20260809 20:27:06.557181 27543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:06 dist-test-slave-rnrw krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307226, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:06.566037 27543 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:27:06.567754 27543 webserver.cc:460] Webserver started at http://127.12.78.2:42611/ using document root <none> and password file <none>
I20260809 20:27:06.568457 27543 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:06.568639 27543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:06.568949 27543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:27:06.572188 27543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "549192f2a5dd4ef8b3e27b28d6a1de49"
format_stamp: "Formatted at 2026-08-09 20:27:06 on dist-test-slave-rnrw"
server_key: "1eb5f9eca6bef5b14e3278062c431c81"
server_key_iv: "6fd96d274bbbc48ac1a3be001fbbe92b"
server_key_version: "encryptionkey@0"
I20260809 20:27:06.573067 27543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "549192f2a5dd4ef8b3e27b28d6a1de49"
format_stamp: "Formatted at 2026-08-09 20:27:06 on dist-test-slave-rnrw"
server_key: "1eb5f9eca6bef5b14e3278062c431c81"
server_key_iv: "6fd96d274bbbc48ac1a3be001fbbe92b"
server_key_version: "encryptionkey@0"
I20260809 20:27:06.578269 27543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:27:06.581729 27558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:06.583076 27543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:27:06.583251 27543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "549192f2a5dd4ef8b3e27b28d6a1de49"
format_stamp: "Formatted at 2026-08-09 20:27:06 on dist-test-slave-rnrw"
server_key: "1eb5f9eca6bef5b14e3278062c431c81"
server_key_iv: "6fd96d274bbbc48ac1a3be001fbbe92b"
server_key_version: "encryptionkey@0"
I20260809 20:27:06.583448 27543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:06.594496 27543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:06.598708 27543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:06.599000 27543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:06.599956 27543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:06.601346 27543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:06.601419 27543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:06.601586 27543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:06.601651 27543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:06.635556 27543 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:38375
I20260809 20:27:06.635689 27677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:38375 every 8 connection(s)
I20260809 20:27:06.636909 27543 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:27:06.638530 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 27543
I20260809 20:27:06.638727 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:27:06.639189 12600 external_mini_cluster.cc:1428] Setting key 349fd3c68c94df9b6418522c066936ab
Aug 09 20:27:06 dist-test-slave-rnrw krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307226, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:06.659786 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:06.639662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:47217 (local address 127.12.78.62:40439)
0809 20:27:06.640180 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:06.640185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:06.643088 (+ 2903us) server_negotiation.cc:407] Connection header received
0809 20:27:06.643143 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:06.643147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:06.643205 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:06.643289 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:06.645424 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:06.646493 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:06.649057 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:06.649723 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:06.649845 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:06.654056 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:06.654061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:06.654065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:06.654099 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:06.656714 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:06.657430 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:06.657436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:06.657444 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:06.657581 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:06.658017 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:06.658021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:06.658023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:06.658295 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:06.658452 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:06.659148 (+ 696us) server_negotiation.cc:299] Negotiation successful
0809 20:27:06.659376 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":290,"threads_started":1}
I20260809 20:27:06.660779 27681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:06.640241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:47217)
0809 20:27:06.641110 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:06.641161 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:06.642814 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:06.643598 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:06.643612 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:06.644385 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:27:06.645239 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:06.645266 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:06.646667 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:06.646675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:06.648912 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:06.648930 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:06.649949 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:06.649959 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:06.650285 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:06.651059 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:27:06.651092 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:27:06.653790 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:27:06.656847 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:06.656856 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:06.656873 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:06.657298 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:06.657693 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:06.657698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:06.657702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:06.657891 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:06.658430 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:27:06.658437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:27:06.658783 (+ 346us) client_negotiation.cc:769] Sending connection context
0809 20:27:06.659055 (+ 272us) client_negotiation.cc:240] Negotiation successful
0809 20:27:06.659595 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":340,"threads_started":1}
I20260809 20:27:06.662385 27678 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40439
I20260809 20:27:06.662705 27678 heartbeater.cc:464] Registering TS with master...
I20260809 20:27:06.663419 27678 heartbeater.cc:511] Master 127.12.78.62:40439 requested a full tablet report, sending...
I20260809 20:27:06.665035 27338 ts_manager.cc:194] Registered new tserver with Master: 549192f2a5dd4ef8b3e27b28d6a1de49 (127.12.78.2:38375)
I20260809 20:27:06.665853 27338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:47217
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:27:06.702975 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40439
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:27:07.046885 27686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:07.047190 27686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:07.047343 27686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:07.054807 27686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:07.054935 27686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:07.055111 27686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:27:07.065032 27686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:07.066433 27686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:07.067907 27686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:07.082029 27694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:07.082093 27691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:07.082698 27692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:07.082959 27686 server_base.cc:1034] running on GCE node
I20260809 20:27:07.083642 27686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:07.084736 27686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:07.085978 27686 hybrid_clock.cc:648] HybridClock initialized: now 1786307227085894 us; error 75 us; skew 500 ppm
I20260809 20:27:07.086313 27686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:07 dist-test-slave-rnrw krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307227, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:07.089877 27686 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:27:07.091491 27686 webserver.cc:460] Webserver started at http://127.12.78.3:38523/ using document root <none> and password file <none>
I20260809 20:27:07.092291 27686 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:07.092435 27686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:07.092797 27686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:27:07.095746 27686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "847579d6597946b29c17345d6f0de37d"
format_stamp: "Formatted at 2026-08-09 20:27:07 on dist-test-slave-rnrw"
server_key: "15737e438a8ba4c53bd37e7833688652"
server_key_iv: "8d12daff831c455eeed431d778f9f018"
server_key_version: "encryptionkey@0"
I20260809 20:27:07.096599 27686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "847579d6597946b29c17345d6f0de37d"
format_stamp: "Formatted at 2026-08-09 20:27:07 on dist-test-slave-rnrw"
server_key: "15737e438a8ba4c53bd37e7833688652"
server_key_iv: "8d12daff831c455eeed431d778f9f018"
server_key_version: "encryptionkey@0"
I20260809 20:27:07.102025 27686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 20:27:07.105702 27701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:07.107097 27686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 20:27:07.107288 27686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "847579d6597946b29c17345d6f0de37d"
format_stamp: "Formatted at 2026-08-09 20:27:07 on dist-test-slave-rnrw"
server_key: "15737e438a8ba4c53bd37e7833688652"
server_key_iv: "8d12daff831c455eeed431d778f9f018"
server_key_version: "encryptionkey@0"
I20260809 20:27:07.107497 27686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:07.128070 27537 heartbeater.cc:503] Master 127.12.78.62:40439 was elected leader, sending a full tablet report...
I20260809 20:27:07.131174 27686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:07.135730 27686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:07.136039 27686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:07.137141 27686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:07.138564 27686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:07.138643 27686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:07.138795 27686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:07.138847 27686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:07.174415 27686 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:46739
I20260809 20:27:07.174476 27818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:46739 every 8 connection(s)
I20260809 20:27:07.175793 27686 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:27:07.180960 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 27686
I20260809 20:27:07.181169 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:27:07.181618 12600 external_mini_cluster.cc:1428] Setting key 3f595469a0a18eef11f954521942ac78
Aug 09 20:27:07 dist-test-slave-rnrw krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307227, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:07.196604 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.178464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:49055 (local address 127.12.78.62:40439)
0809 20:27:07.178948 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:07.178952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:07.179825 (+ 873us) server_negotiation.cc:407] Connection header received
0809 20:27:07.181897 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:07.181903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:07.181998 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:07.182126 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:07.183713 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.184608 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.186216 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.186625 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.186765 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.190456 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:07.190461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:07.190465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:07.190503 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:07.193204 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:07.194008 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:07.194014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:07.194022 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:07.194156 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:07.194618 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:07.194624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:07.194627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:07.194927 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:07.195109 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:07.195873 (+ 764us) server_negotiation.cc:299] Negotiation successful
0809 20:27:07.196132 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":263,"threads_started":1}
I20260809 20:27:07.197726 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.178925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:49055)
0809 20:27:07.179651 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:07.179698 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:07.181592 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:07.182376 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:07.182384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:07.182918 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:27:07.183519 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:07.183536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.184775 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.184779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:07.186046 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:07.186058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.186857 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.186865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:07.187131 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.187806 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:27:07.187829 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:27:07.190204 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:27:07.193365 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:07.193374 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:07.193394 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:07.193873 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:07.194275 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:07.194281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:07.194284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:07.194495 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:07.195075 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:27:07.195084 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:27:07.195489 (+ 405us) client_negotiation.cc:769] Sending connection context
0809 20:27:07.195786 (+ 297us) client_negotiation.cc:240] Negotiation successful
0809 20:27:07.196419 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":286,"threads_started":1}
I20260809 20:27:07.199234 27819 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40439
I20260809 20:27:07.199556 27819 heartbeater.cc:464] Registering TS with master...
I20260809 20:27:07.200380 27819 heartbeater.cc:511] Master 127.12.78.62:40439 requested a full tablet report, sending...
I20260809 20:27:07.201947 27338 ts_manager.cc:194] Registered new tserver with Master: 847579d6597946b29c17345d6f0de37d (127.12.78.3:46739)
I20260809 20:27:07.202828 27338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:49055
I20260809 20:27:07.209877 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:27:07.229266 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.215375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.12.78.62:40439)
0809 20:27:07.215628 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:07.215636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:07.215976 (+ 340us) server_negotiation.cc:407] Connection header received
0809 20:27:07.216146 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:07.216150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:07.216224 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:07.216312 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:07.217964 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.218734 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.220217 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.220731 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.220835 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.223554 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:07.223559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:07.223562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:07.223603 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:07.226488 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:07.227238 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:07.227243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:07.227245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:07.227373 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:07.227771 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:07.227774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:07.227776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:07.228111 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:07.228301 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:07.228741 (+ 440us) server_negotiation.cc:299] Negotiation successful
0809 20:27:07.228933 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 20:27:07.233664 27338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:27:07.240913 27338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:27:07.278746 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.254433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46056 (local address 127.12.78.2:38375)
0809 20:27:07.268052 (+ 13619us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:07.268065 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:07.268106 (+ 41us) server_negotiation.cc:407] Connection header received
0809 20:27:07.268243 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:07.268252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:07.268605 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:07.268836 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:07.270032 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.271877 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.275617 (+ 3740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.277391 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.277554 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.277766 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:07.278192 (+ 426us) server_negotiation.cc:299] Negotiation successful
0809 20:27:07.278393 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13427,"thread_start_us":13293,"threads_started":1}
I20260809 20:27:07.278867 27834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.254170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:38375 (local address 127.0.0.1:46056)
0809 20:27:07.266899 (+ 12729us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:07.266980 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:07.267289 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:07.269166 (+ 1877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:07.269175 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:07.269208 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:07.269786 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:07.269799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.272083 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.272089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:07.275399 (+ 3310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:07.275416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.277643 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.277650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:07.277867 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.277932 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 20:27:07.278101 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 20:27:07.278343 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12321,"mutex_wait_us":165,"thread_start_us":13955,"threads_started":1}
I20260809 20:27:07.279136 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.254412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:46619 (local address 127.0.0.1:44204)
0809 20:27:07.268409 (+ 13997us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:07.268433 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:07.268739 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:07.269786 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:07.269791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:07.269818 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:07.270303 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:07.270316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.272482 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.272493 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:07.275399 (+ 2906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:07.275415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.277975 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.277984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:07.278281 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.278316 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 20:27:07.278415 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 20:27:07.278636 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13867,"mutex_wait_us":117,"thread_start_us":11887,"threads_started":1}
I20260809 20:27:07.280798 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.269919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:46739 (local address 127.0.0.1:57150)
0809 20:27:07.272218 (+ 2299us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:07.272237 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:07.272366 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:07.273521 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:07.273525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:07.273552 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:07.274099 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:07.274108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.275897 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.275902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:07.278039 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:07.278054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.279887 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.279893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:07.280093 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.280126 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 20:27:07.280221 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 20:27:07.280369 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2135,"mutex_wait_us":1,"thread_start_us":2540,"threads_started":1}
I20260809 20:27:07.280855 27836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.254706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.12.78.1:46619)
0809 20:27:07.265753 (+ 11047us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:07.265769 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:07.268606 (+ 2837us) server_negotiation.cc:407] Connection header received
0809 20:27:07.268964 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:07.268973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:07.269227 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:07.269436 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:07.270562 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.272288 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.275620 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.277399 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.277542 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.277804 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:07.280255 (+ 2451us) server_negotiation.cc:299] Negotiation successful
0809 20:27:07.280451 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10705,"thread_start_us":11068,"threads_started":1}
I20260809 20:27:07.282220 27837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.268147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.12.78.3:46739)
0809 20:27:07.272954 (+ 4807us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:07.272966 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:07.272998 (+ 32us) server_negotiation.cc:407] Connection header received
0809 20:27:07.273100 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:07.273110 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:07.273326 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:07.273532 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:07.274292 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.275721 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.278233 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.279677 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.279785 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.279958 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:07.281647 (+ 1689us) server_negotiation.cc:299] Negotiation successful
0809 20:27:07.281892 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4615,"thread_start_us":3335,"threads_started":1}
I20260809 20:27:07.283512 27612 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6c56bea90a4f00b872bbbe0f238079 (DEFAULT_TABLE table=test-table [id=246485b1ad304f7b9f1bd45071aac083]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:27:07.285148 27612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6c56bea90a4f00b872bbbe0f238079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:07.284730 27467 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6c56bea90a4f00b872bbbe0f238079 (DEFAULT_TABLE table=test-table [id=246485b1ad304f7b9f1bd45071aac083]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:27:07.285135 27750 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6c56bea90a4f00b872bbbe0f238079 (DEFAULT_TABLE table=test-table [id=246485b1ad304f7b9f1bd45071aac083]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:27:07.286195 27467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6c56bea90a4f00b872bbbe0f238079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:07.286617 27750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6c56bea90a4f00b872bbbe0f238079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:07.298053 27839 tablet_bootstrap.cc:492] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d: Bootstrap starting.
I20260809 20:27:07.302213 27839 tablet_bootstrap.cc:654] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:07.303180 27841 tablet_bootstrap.cc:492] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49: Bootstrap starting.
I20260809 20:27:07.303697 27839 log.cc:824] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:07.305541 27840 tablet_bootstrap.cc:492] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec: Bootstrap starting.
I20260809 20:27:07.307260 27839 tablet_bootstrap.cc:492] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d: No bootstrap required, opened a new log
I20260809 20:27:07.307520 27841 tablet_bootstrap.cc:654] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:07.307669 27839 ts_tablet_manager.cc:1397] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260809 20:27:07.308949 27841 log.cc:824] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:07.309613 27840 tablet_bootstrap.cc:654] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:07.311401 27840 log.cc:824] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:07.316278 27839 raft_consensus.cc:348] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:07.316946 27839 raft_consensus.cc:374] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:07.317076 27839 raft_consensus.cc:729] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847579d6597946b29c17345d6f0de37d, State: Initialized, Role: FOLLOWER
I20260809 20:27:07.317759 27839 consensus_queue.cc:260] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:07.318904 27841 tablet_bootstrap.cc:492] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49: No bootstrap required, opened a new log
I20260809 20:27:07.318919 27819 heartbeater.cc:503] Master 127.12.78.62:40439 was elected leader, sending a full tablet report...
I20260809 20:27:07.319242 27841 ts_tablet_manager.cc:1397] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20260809 20:27:07.319582 27839 ts_tablet_manager.cc:1428] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260809 20:27:07.320818 27840 tablet_bootstrap.cc:492] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec: No bootstrap required, opened a new log
I20260809 20:27:07.321098 27840 ts_tablet_manager.cc:1397] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260809 20:27:07.328944 27840 raft_consensus.cc:348] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:07.329093 27841 raft_consensus.cc:348] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:07.329391 27840 raft_consensus.cc:374] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:07.329519 27840 raft_consensus.cc:729] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8370621deb3d465d840273954440c8ec, State: Initialized, Role: FOLLOWER
I20260809 20:27:07.329555 27841 raft_consensus.cc:374] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:07.329694 27841 raft_consensus.cc:729] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549192f2a5dd4ef8b3e27b28d6a1de49, State: Initialized, Role: FOLLOWER
I20260809 20:27:07.330169 27840 consensus_queue.cc:260] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:07.330273 27841 consensus_queue.cc:260] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:07.331501 27678 heartbeater.cc:503] Master 127.12.78.62:40439 was elected leader, sending a full tablet report...
I20260809 20:27:07.332213 27840 ts_tablet_manager.cc:1428] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260809 20:27:07.332226 27841 ts_tablet_manager.cc:1428] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260809 20:27:07.342761 27538 tablet.cc:2369] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:27:07.390378 27679 tablet.cc:2369] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:27:07.427531 27820 tablet.cc:2369] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:27:07.488783 27846 raft_consensus.cc:482] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:27:07.489143 27846 raft_consensus.cc:504] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:07.490947 27846 leader_election.cc:288] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 549192f2a5dd4ef8b3e27b28d6a1de49 (127.12.78.2:38375), 847579d6597946b29c17345d6f0de37d (127.12.78.3:46739)
I20260809 20:27:07.498355 27837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.491516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:40265 (local address 127.12.78.3:46739)
0809 20:27:07.491758 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:07.491768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:07.492523 (+ 755us) server_negotiation.cc:407] Connection header received
0809 20:27:07.492836 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:07.492840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:07.492901 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:07.492999 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:07.493643 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.494392 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.496422 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.497282 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.497444 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.497558 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:07.497910 (+ 352us) server_negotiation.cc:299] Negotiation successful
0809 20:27:07.498068 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 20:27:07.498358 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.491466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:49379 (local address 127.12.78.2:38375)
0809 20:27:07.491720 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:07.491729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:07.492578 (+ 849us) server_negotiation.cc:407] Connection header received
0809 20:27:07.492835 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:07.492844 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:07.492904 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:07.492992 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:07.493695 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.494391 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.496414 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.497274 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.497424 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.497553 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:07.497920 (+ 367us) server_negotiation.cc:299] Negotiation successful
0809 20:27:07.498060 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 20:27:07.498564 27849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.491450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:38375 (local address 127.12.78.1:49379)
0809 20:27:07.492434 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:07.492449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:07.492695 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:07.493027 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:07.493032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:07.493064 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:07.493515 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:07.493526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.494527 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.494535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:07.496260 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:07.496272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.497572 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.497581 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:07.497766 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.497789 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 20:27:07.497890 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 20:27:07.498079 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":147,"spinlock_wait_cycles":108544,"thread_start_us":597,"threads_started":1}
I20260809 20:27:07.498587 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.491222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:46739 (local address 127.12.78.1:40265)
0809 20:27:07.492384 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:07.492400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:07.492674 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:07.493020 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:07.493024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:07.493049 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:07.493477 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:07.493490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.494529 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.494535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:07.496266 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:07.496278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.497527 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.497533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:07.497704 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.497771 (+ 67us) client_negotiation.cc:769] Sending connection context
0809 20:27:07.497884 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 20:27:07.498091 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":144,"thread_start_us":798,"threads_started":1}
I20260809 20:27:07.499666 27632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6c56bea90a4f00b872bbbe0f238079" candidate_uuid: "8370621deb3d465d840273954440c8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" is_pre_election: true
I20260809 20:27:07.499662 27771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6c56bea90a4f00b872bbbe0f238079" candidate_uuid: "8370621deb3d465d840273954440c8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847579d6597946b29c17345d6f0de37d" is_pre_election: true
I20260809 20:27:07.500171 27771 raft_consensus.cc:2454] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8370621deb3d465d840273954440c8ec in term 0.
I20260809 20:27:07.500171 27632 raft_consensus.cc:2454] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8370621deb3d465d840273954440c8ec in term 0.
I20260809 20:27:07.500880 27419 leader_election.cc:302] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 549192f2a5dd4ef8b3e27b28d6a1de49, 8370621deb3d465d840273954440c8ec, 847579d6597946b29c17345d6f0de37d; no voters:
I20260809 20:27:07.501310 27846 raft_consensus.cc:2781] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:27:07.501478 27846 raft_consensus.cc:482] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:27:07.501590 27846 raft_consensus.cc:3037] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:07.503868 27846 raft_consensus.cc:504] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:07.504619 27846 leader_election.cc:288] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [CANDIDATE]: Term 1 election: Requested vote from peers 549192f2a5dd4ef8b3e27b28d6a1de49 (127.12.78.2:38375), 847579d6597946b29c17345d6f0de37d (127.12.78.3:46739)
I20260809 20:27:07.505200 27771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6c56bea90a4f00b872bbbe0f238079" candidate_uuid: "8370621deb3d465d840273954440c8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847579d6597946b29c17345d6f0de37d"
I20260809 20:27:07.505189 27632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6c56bea90a4f00b872bbbe0f238079" candidate_uuid: "8370621deb3d465d840273954440c8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49"
I20260809 20:27:07.505447 27771 raft_consensus.cc:3037] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:07.505455 27632 raft_consensus.cc:3037] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:07.508064 27771 raft_consensus.cc:2454] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8370621deb3d465d840273954440c8ec in term 1.
I20260809 20:27:07.508066 27632 raft_consensus.cc:2454] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8370621deb3d465d840273954440c8ec in term 1.
I20260809 20:27:07.508574 27418 leader_election.cc:302] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549192f2a5dd4ef8b3e27b28d6a1de49, 8370621deb3d465d840273954440c8ec; no voters:
I20260809 20:27:07.508986 27846 raft_consensus.cc:2781] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:27:07.509479 27846 raft_consensus.cc:686] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [term 1 LEADER]: Becoming Leader. State: Replica: 8370621deb3d465d840273954440c8ec, State: Running, Role: LEADER
I20260809 20:27:07.510038 27846 consensus_queue.cc:237] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:07.514899 27338 catalog_manager.cc:5495] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 8370621deb3d465d840273954440c8ec (127.12.78.1). New cstate: current_term: 1 leader_uuid: "8370621deb3d465d840273954440c8ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } health_report { overall_health: UNKNOWN } } }
I20260809 20:27:07.578896 27836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.573125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.12.78.1:46619)
0809 20:27:07.573308 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:07.573312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:07.573330 (+ 18us) server_negotiation.cc:407] Connection header received
0809 20:27:07.573517 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:07.573521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:07.573630 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:07.573762 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:27:07.574128 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.574884 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.576461 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.577016 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.577135 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.577743 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:27:07.578265 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:27:07.578380 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:07.578486 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 20:27:07.578588 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 20:27:07.587747 27771 raft_consensus.cc:1261] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d [term 1 FOLLOWER]: Refusing update from remote peer 8370621deb3d465d840273954440c8ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:27:07.587749 27632 raft_consensus.cc:1261] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 1 FOLLOWER]: Refusing update from remote peer 8370621deb3d465d840273954440c8ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:27:07.588900 27850 consensus_queue.cc:1035] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:27:07.589248 27846 consensus_queue.cc:1035] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:27:07.617587 27338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:27:07.617873 27338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:27:07.621366 27338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Sending DeleteTablet for 3 replicas of tablet 2e6c56bea90a4f00b872bbbe0f238079
I20260809 20:27:07.622525 27612 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6c56bea90a4f00b872bbbe0f238079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:27:07 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:46056
I20260809 20:27:07.622706 27467 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6c56bea90a4f00b872bbbe0f238079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:27:07 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:44204
I20260809 20:27:07.623386 27750 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6c56bea90a4f00b872bbbe0f238079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:27:07 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:57150
I20260809 20:27:07.623983 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 27401
I20260809 20:27:07.639791 27866 tablet_replica.cc:330] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec: stopping tablet replica
I20260809 20:27:07.640460 27866 raft_consensus.cc:2229] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:27:07.642295 27865 tablet_replica.cc:330] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49: stopping tablet replica
I20260809 20:27:07.642728 27865 raft_consensus.cc:2229] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:27:07.643314 27865 raft_consensus.cc:2258] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:27:07.643730 27866 raft_consensus.cc:2258] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:27:07.646152 27866 ts_tablet_manager.cc:1905] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:27:07.650247 27865 ts_tablet_manager.cc:1905] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:27:07.655977 27866 ts_tablet_manager.cc:1918] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:27:07.656283 27866 log.cc:1196] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/2e6c56bea90a4f00b872bbbe0f238079
I20260809 20:27:07.657140 27866 ts_tablet_manager.cc:1939] T 2e6c56bea90a4f00b872bbbe0f238079 P 8370621deb3d465d840273954440c8ec: Deleting consensus metadata
I20260809 20:27:07.657666 27865 ts_tablet_manager.cc:1918] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:27:07.657877 27865 log.cc:1196] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/2e6c56bea90a4f00b872bbbe0f238079
I20260809 20:27:07.658488 27865 ts_tablet_manager.cc:1939] T 2e6c56bea90a4f00b872bbbe0f238079 P 549192f2a5dd4ef8b3e27b28d6a1de49: Deleting consensus metadata
I20260809 20:27:07.659616 27323 catalog_manager.cc:4841] TS 8370621deb3d465d840273954440c8ec (127.12.78.1:46619): tablet 2e6c56bea90a4f00b872bbbe0f238079 (table test-table [id=246485b1ad304f7b9f1bd45071aac083]) successfully deleted
I20260809 20:27:07.660266 27322 catalog_manager.cc:4841] TS 549192f2a5dd4ef8b3e27b28d6a1de49 (127.12.78.2:38375): tablet 2e6c56bea90a4f00b872bbbe0f238079 (table test-table [id=246485b1ad304f7b9f1bd45071aac083]) successfully deleted
I20260809 20:27:07.661091 27867 tablet_replica.cc:330] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d: stopping tablet replica
I20260809 20:27:07.661612 27867 raft_consensus.cc:2229] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:27:07.662155 27867 raft_consensus.cc:2258] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:27:07.665781 27867 ts_tablet_manager.cc:1905] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 20:27:07 dist-test-slave-rnrw krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:27:07.674546 27867 ts_tablet_manager.cc:1918] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:27:07.674865 27867 log.cc:1196] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/2e6c56bea90a4f00b872bbbe0f238079
I20260809 20:27:07.675595 27867 ts_tablet_manager.cc:1939] T 2e6c56bea90a4f00b872bbbe0f238079 P 847579d6597946b29c17345d6f0de37d: Deleting consensus metadata
I20260809 20:27:07.677968 27323 catalog_manager.cc:4841] TS 847579d6597946b29c17345d6f0de37d (127.12.78.3:46739): tablet 2e6c56bea90a4f00b872bbbe0f238079 (table test-table [id=246485b1ad304f7b9f1bd45071aac083]) successfully deleted
I20260809 20:27:07.679585 27836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.625742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.12.78.1:46619)
0809 20:27:07.626035 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:07.626045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:07.632494 (+ 6449us) server_negotiation.cc:407] Connection header received
0809 20:27:07.633851 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:07.633861 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:07.633982 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:07.634097 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:07.657202 (+ 23105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.658606 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.663817 (+ 5211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.664533 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.664722 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.671073 (+ 6351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:07.671093 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:07.671113 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:07.671167 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:07.675502 (+ 4335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:07.676706 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:07.676716 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:07.676719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:07.676919 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:07.677449 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:07.677463 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:07.677470 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:07.678038 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:07.678407 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:07.678895 (+ 488us) server_negotiation.cc:299] Negotiation successful
0809 20:27:07.679201 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 20:27:07.680387 27532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:07.787689 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 27401
I20260809 20:27:07.809293 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 27543
Aug 09 20:27:07 dist-test-slave-rnrw krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:27:07.825603 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.810057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.12.78.2:38375)
0809 20:27:07.810231 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:07.810236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:07.810286 (+ 50us) server_negotiation.cc:407] Connection header received
0809 20:27:07.810492 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:07.810496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:07.810579 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:07.810668 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:07.812605 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.813420 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.814754 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.815130 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.815224 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.819473 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:07.819487 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:07.819504 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:07.819569 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:07.822790 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:07.823564 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:07.823572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:07.823575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:07.823727 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:07.824077 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:07.824081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:07.824083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:07.824618 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:07.824892 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:07.825126 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 20:27:07.825311 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 20:27:07.826408 27673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:07.917842 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 27543
I20260809 20:27:07.937405 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 27686
Aug 09 20:27:07 dist-test-slave-rnrw krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:27:07.953024 27837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:07.938470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.12.78.3:46739)
0809 20:27:07.938710 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:07.938717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:07.938739 (+ 22us) server_negotiation.cc:407] Connection header received
0809 20:27:07.938911 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:07.938917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:07.939031 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:07.939143 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:07.940491 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.941275 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.942636 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:07.943016 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:07.943118 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:07.947793 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:07.947810 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:07.947827 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:07.947863 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:07.950487 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:07.951157 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:07.951164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:07.951167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:07.951322 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:07.951632 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:07.951635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:07.951637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:07.951978 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:07.952220 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:07.952466 (+ 246us) server_negotiation.cc:299] Negotiation successful
0809 20:27:07.952701 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 20:27:07.953699 27813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:08.044370 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 27686
I20260809 20:27:08.064085 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 27303
I20260809 20:27:08.065508 27368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:08.178706 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 27303
I20260809 20:27:08.203094 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40439
--webserver_interface=127.12.78.62
--webserver_port=44707
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:40439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:27:08.572253 27873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:08.572692 27873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:08.572853 27873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:08.580919 27873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:27:08.581082 27873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:08.581193 27873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:08.581296 27873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:27:08.581372 27873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:27:08.592140 27873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:40439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=44707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:08.593681 27873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:08.595089 27873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:08.620169 27881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:08.620214 27884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:08.620247 27880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:08.621135 27873 server_base.cc:1034] running on GCE node
I20260809 20:27:08.622056 27873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:08.623796 27873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:08.625057 27873 hybrid_clock.cc:648] HybridClock initialized: now 1786307228624977 us; error 59 us; skew 500 ppm
I20260809 20:27:08.625561 27873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:08 dist-test-slave-rnrw krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307228, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:08.629667 27873 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:27:08.631639 27873 webserver.cc:460] Webserver started at http://127.12.78.62:44707/ using document root <none> and password file <none>
I20260809 20:27:08.632508 27873 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:08.632737 27873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:08.639392 27873 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 20:27:08.642949 27891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:08.646001 27873 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260809 20:27:08.646251 27873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "0fff14088a4e4dcdae437f3723410567"
format_stamp: "Formatted at 2026-08-09 20:27:05 on dist-test-slave-rnrw"
server_key: "ce18f090a1f299fb4aea233f59cef1ad"
server_key_iv: "bbd8c6dc22f6152b71638c6a7670b5ae"
server_key_version: "encryptionkey@0"
I20260809 20:27:08.647465 27873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:08.662830 27873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:08.668016 27873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:08.668274 27873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:08.687649 27873 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:40439
I20260809 20:27:08.687672 27943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:40439 every 8 connection(s)
I20260809 20:27:08.689314 27873 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:27:08.691597 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 27873
I20260809 20:27:08.692426 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:46619
--local_ip_for_outbound_sockets=127.12.78.1
--tserver_master_addrs=127.12.78.62:40439
--webserver_port=45049
--webserver_interface=127.12.78.1
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:27:08.697398 27944 sys_catalog.cc:263] Verifying existing consensus state
I20260809 20:27:08.701647 27944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Bootstrap starting.
I20260809 20:27:08.724128 27944 log.cc:824] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:08.737743 27944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 20:27:08.738358 27944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Bootstrap complete.
I20260809 20:27:08.748950 27944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } }
I20260809 20:27:08.749938 27944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fff14088a4e4dcdae437f3723410567, State: Initialized, Role: FOLLOWER
I20260809 20:27:08.750658 27944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } }
I20260809 20:27:08.750931 27944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:27:08.751106 27944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:27:08.751297 27944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 1 FOLLOWER]: Advancing to term 2
I20260809 20:27:08.755792 27944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } }
I20260809 20:27:08.756373 27944 leader_election.cc:302] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fff14088a4e4dcdae437f3723410567; no voters:
I20260809 20:27:08.757126 27944 leader_election.cc:288] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 20:27:08.757284 27949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 20:27:08.758558 27949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 2 LEADER]: Becoming Leader. State: Replica: 0fff14088a4e4dcdae437f3723410567, State: Running, Role: LEADER
I20260809 20:27:08.759291 27949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } }
I20260809 20:27:08.759768 27944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:27:08.762259 27951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fff14088a4e4dcdae437f3723410567. Latest consensus state: current_term: 2 leader_uuid: "0fff14088a4e4dcdae437f3723410567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } } }
I20260809 20:27:08.762619 27951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: This master's current role is: LEADER
I20260809 20:27:08.762554 27950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0fff14088a4e4dcdae437f3723410567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } } }
I20260809 20:27:08.762991 27950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: This master's current role is: LEADER
I20260809 20:27:08.764098 27956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:27:08.783681 27956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:27:08.789037 27956 catalog_manager.cc:1259] Loaded cluster ID: 08dc62be39534972a5ce404cd5d9f274
I20260809 20:27:08.789222 27956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:27:08.796000 27956 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 20:27:08.812820 27956 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 20:27:08.822384 27956 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:27:08.824433 27956 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Loaded TSK: 0
I20260809 20:27:08.826036 27956 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 20:27:09.068809 27946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:09.069105 27946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:09.069249 27946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:09.076900 27946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:09.077018 27946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:09.077189 27946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:27:09.087428 27946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:46619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=45049
--tserver_master_addrs=127.12.78.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:09.088896 27946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:09.090296 27946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:09.105973 27975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:09.106249 27978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:09.106542 27976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:09.106902 27946 server_base.cc:1034] running on GCE node
I20260809 20:27:09.107592 27946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:09.108659 27946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:09.109958 27946 hybrid_clock.cc:648] HybridClock initialized: now 1786307229109888 us; error 68 us; skew 500 ppm
I20260809 20:27:09.110298 27946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:09 dist-test-slave-rnrw krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307229, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:09.113665 27946 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:27:09.115873 27946 webserver.cc:460] Webserver started at http://127.12.78.1:45049/ using document root <none> and password file <none>
I20260809 20:27:09.116699 27946 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:09.116865 27946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:09.122887 27946 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 20:27:09.126160 27985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:09.127789 27946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:27:09.127983 27946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "8370621deb3d465d840273954440c8ec"
format_stamp: "Formatted at 2026-08-09 20:27:05 on dist-test-slave-rnrw"
server_key: "70b9dbf59576ea10709938359335128d"
server_key_iv: "b56599bbd1a9c7c1b5940d491e1f316e"
server_key_version: "encryptionkey@0"
I20260809 20:27:09.129324 27946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:09.145154 27946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:09.149987 27946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:09.150310 27946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:09.151513 27946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:09.153237 27946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:09.153355 27946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:09.153498 27946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:09.153599 27946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:09.192497 27946 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:46619
I20260809 20:27:09.192605 28100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:46619 every 8 connection(s)
I20260809 20:27:09.194130 27946 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:27:09.197125 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 27946
I20260809 20:27:09.198043 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:38375
--local_ip_for_outbound_sockets=127.12.78.2
--tserver_master_addrs=127.12.78.62:40439
--webserver_port=42611
--webserver_interface=127.12.78.2
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 20:27:09 dist-test-slave-rnrw krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307229, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:09.218938 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:09.197103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:49933 (local address 127.12.78.62:40439)
0809 20:27:09.197818 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:09.197848 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:09.198604 (+ 756us) server_negotiation.cc:407] Connection header received
0809 20:27:09.200776 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:09.200788 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:09.201118 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:09.201472 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:09.203074 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:09.204344 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:09.205980 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:09.206652 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:09.206868 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:09.210875 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:09.210883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:09.210897 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:09.210936 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:09.214027 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:09.215136 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:09.215144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:09.215149 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:09.215321 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:09.215838 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:09.215841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:09.215843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:09.216351 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:09.216506 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:09.217501 (+ 995us) server_negotiation.cc:299] Negotiation successful
0809 20:27:09.217844 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":353,"threads_started":1}
I20260809 20:27:09.219365 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:09.197490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:49933)
0809 20:27:09.198414 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:09.198460 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:09.200190 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:09.201618 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:09.201631 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:09.202283 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:27:09.202833 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:09.202850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:09.204530 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:09.204534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:09.205822 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:09.205840 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:09.206894 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:09.206902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:09.207184 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:09.207921 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:27:09.207945 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:27:09.210583 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:27:09.214196 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:09.214209 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:09.214228 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:09.214967 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:09.215439 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:09.215446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:09.215452 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:09.215722 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:09.216496 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:27:09.216507 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:27:09.216987 (+ 480us) client_negotiation.cc:769] Sending connection context
0809 20:27:09.217309 (+ 322us) client_negotiation.cc:240] Negotiation successful
0809 20:27:09.217950 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":340,"threads_started":1}
I20260809 20:27:09.222575 28101 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40439
I20260809 20:27:09.222985 28101 heartbeater.cc:464] Registering TS with master...
I20260809 20:27:09.224015 28101 heartbeater.cc:511] Master 127.12.78.62:40439 requested a full tablet report, sending...
I20260809 20:27:09.226637 27909 ts_manager.cc:194] Registered new tserver with Master: 8370621deb3d465d840273954440c8ec (127.12.78.1:46619)
I20260809 20:27:09.228119 27909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:49933
W20260809 20:27:09.544992 28105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:09.545325 28105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:09.545502 28105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:09.552959 28105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:09.553084 28105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:09.553246 28105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:27:09.563390 28105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:38375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=42611
--tserver_master_addrs=127.12.78.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:09.564827 28105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:09.566341 28105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:09.580065 28111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:09.580107 28114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:09.580029 28112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:09.581494 28105 server_base.cc:1034] running on GCE node
I20260809 20:27:09.582162 28105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:09.583194 28105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:09.584421 28105 hybrid_clock.cc:648] HybridClock initialized: now 1786307229584360 us; error 52 us; skew 500 ppm
I20260809 20:27:09.584784 28105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:09 dist-test-slave-rnrw krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307229, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:09.588073 28105 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:27:09.589627 28105 webserver.cc:460] Webserver started at http://127.12.78.2:42611/ using document root <none> and password file <none>
I20260809 20:27:09.590340 28105 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:09.590461 28105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:09.596079 28105 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 20:27:09.598901 28122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:09.600256 28105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 20:27:09.600402 28105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "549192f2a5dd4ef8b3e27b28d6a1de49"
format_stamp: "Formatted at 2026-08-09 20:27:06 on dist-test-slave-rnrw"
server_key: "1eb5f9eca6bef5b14e3278062c431c81"
server_key_iv: "6fd96d274bbbc48ac1a3be001fbbe92b"
server_key_version: "encryptionkey@0"
I20260809 20:27:09.601442 28105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:09.619912 28105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:09.624265 28105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:09.624511 28105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:09.625612 28105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:09.626991 28105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:09.627089 28105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:09.627175 28105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:09.627238 28105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:09.661804 28105 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:38375
I20260809 20:27:09.661973 28241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:38375 every 8 connection(s)
I20260809 20:27:09.663305 28105 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:27:09.667680 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 28105
I20260809 20:27:09.668514 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:46739
--local_ip_for_outbound_sockets=127.12.78.3
--tserver_master_addrs=127.12.78.62:40439
--webserver_port=38523
--webserver_interface=127.12.78.3
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 20:27:09 dist-test-slave-rnrw krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307229, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:09.689297 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:09.666050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:34635 (local address 127.12.78.62:40439)
0809 20:27:09.666280 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:09.666287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:09.667620 (+ 1333us) server_negotiation.cc:407] Connection header received
0809 20:27:09.670649 (+ 3029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:09.670654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:09.670733 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:09.670830 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:09.673270 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:09.674024 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:09.676685 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:09.677134 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:09.677234 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:09.682227 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:09.682233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:09.682236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:09.682275 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:09.685003 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:09.686079 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:09.686083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:09.686085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:09.686217 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:09.686839 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:09.686845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:09.686848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:09.687297 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:09.687530 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:09.688567 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0809 20:27:09.688863 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 20:27:09.690732 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:09.666544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:34635)
0809 20:27:09.667357 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:09.667411 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:09.670253 (+ 2842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:09.671140 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:09.671154 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:09.672099 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:27:09.673029 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:09.673059 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:09.674220 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:09.674228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:09.676474 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:09.676492 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:09.677373 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:09.677382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:09.677762 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:09.678555 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:27:09.678591 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:27:09.681917 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:27:09.685163 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:09.685176 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:09.685200 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:09.685946 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:09.686328 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:09.686336 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:09.686344 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:09.686642 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:09.687485 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:27:09.687498 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:27:09.688019 (+ 521us) client_negotiation.cc:769] Sending connection context
0809 20:27:09.688372 (+ 353us) client_negotiation.cc:240] Negotiation successful
0809 20:27:09.689085 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":305,"threads_started":1}
I20260809 20:27:09.692754 28242 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40439
I20260809 20:27:09.693149 28242 heartbeater.cc:464] Registering TS with master...
I20260809 20:27:09.694243 28242 heartbeater.cc:511] Master 127.12.78.62:40439 requested a full tablet report, sending...
I20260809 20:27:09.696233 27909 ts_manager.cc:194] Registered new tserver with Master: 549192f2a5dd4ef8b3e27b28d6a1de49 (127.12.78.2:38375)
I20260809 20:27:09.697443 27909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:34635
W20260809 20:27:10.025023 28245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:10.025359 28245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:10.025518 28245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:10.033648 28245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:10.033795 28245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:10.034008 28245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:27:10.044181 28245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:46739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=38523
--tserver_master_addrs=127.12.78.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:10.045583 28245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:10.046962 28245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:10.060665 28251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:10.060725 28253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:10.060632 28250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:10.061643 28245 server_base.cc:1034] running on GCE node
I20260809 20:27:10.062302 28245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:10.063263 28245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:10.064474 28245 hybrid_clock.cc:648] HybridClock initialized: now 1786307230064422 us; error 47 us; skew 500 ppm
I20260809 20:27:10.064849 28245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:10 dist-test-slave-rnrw krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307230, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:10.068013 28245 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:27:10.069540 28245 webserver.cc:460] Webserver started at http://127.12.78.3:38523/ using document root <none> and password file <none>
I20260809 20:27:10.070242 28245 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:10.070405 28245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:10.075753 28245 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260809 20:27:10.078531 28260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:10.079846 28245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 20:27:10.080024 28245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "847579d6597946b29c17345d6f0de37d"
format_stamp: "Formatted at 2026-08-09 20:27:07 on dist-test-slave-rnrw"
server_key: "15737e438a8ba4c53bd37e7833688652"
server_key_iv: "8d12daff831c455eeed431d778f9f018"
server_key_version: "encryptionkey@0"
I20260809 20:27:10.081089 28245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:10.106564 28245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:10.110716 28245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:10.111006 28245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:10.112135 28245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:10.113668 28245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:10.113796 28245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:10.113926 28245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:10.114012 28245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:10.147763 28245 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:46739
I20260809 20:27:10.147851 28380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:46739 every 8 connection(s)
I20260809 20:27:10.149323 28245 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:27:10.153182 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 28245
Aug 09 20:27:10 dist-test-slave-rnrw krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307230, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:10.174993 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.154258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.12.78.62:40439)
0809 20:27:10.154722 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:10.154726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:10.158013 (+ 3287us) server_negotiation.cc:407] Connection header received
0809 20:27:10.158158 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:10.158162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:10.158232 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:10.158346 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:10.159892 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.161006 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.163191 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.163927 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.164066 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.168021 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:10.168028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:10.168032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:10.168074 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:10.171369 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:10.172372 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:10.172375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:10.172378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:10.172500 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:10.173060 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:10.173065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:10.173067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:10.173411 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:10.173712 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:10.174069 (+ 357us) server_negotiation.cc:299] Negotiation successful
0809 20:27:10.174341 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":371,"spinlock_wait_cycles":36224,"thread_start_us":265,"threads_started":1}
I20260809 20:27:10.176313 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.152148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:41695 (local address 127.12.78.62:40439)
0809 20:27:10.152337 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:10.152342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:10.153933 (+ 1591us) server_negotiation.cc:407] Connection header received
0809 20:27:10.157431 (+ 3498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:10.157435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:10.157505 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:10.157665 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:10.159476 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.160629 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.163211 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.163988 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.164161 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.168861 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:10.168866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:10.168886 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:10.168939 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:10.172681 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:10.173668 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:10.173673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:10.173676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:10.173799 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:10.174233 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:10.174238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:10.174242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:10.174599 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:10.174738 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:10.175708 (+ 970us) server_negotiation.cc:299] Negotiation successful
0809 20:27:10.175952 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":65,"spinlock_wait_cycles":9984}
I20260809 20:27:10.177711 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.152697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:41695)
0809 20:27:10.153723 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:10.153779 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:10.156992 (+ 3213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:10.157914 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:10.157923 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:10.158709 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:27:10.159307 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:10.159325 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.160806 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.160810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:10.163064 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:10.163085 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.164237 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.164247 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:10.164639 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.165330 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:27:10.165365 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:27:10.168606 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:27:10.172838 (+ 4232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:10.172849 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:10.172867 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:10.173507 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:10.173909 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:10.173914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:10.173918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:10.174119 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:10.174749 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:27:10.174759 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:27:10.175297 (+ 538us) client_negotiation.cc:769] Sending connection context
0809 20:27:10.175630 (+ 333us) client_negotiation.cc:240] Negotiation successful
0809 20:27:10.176193 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":476,"threads_started":1}
I20260809 20:27:10.179122 28381 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40439
I20260809 20:27:10.179452 28381 heartbeater.cc:464] Registering TS with master...
I20260809 20:27:10.180239 28381 heartbeater.cc:511] Master 127.12.78.62:40439 requested a full tablet report, sending...
I20260809 20:27:10.181715 27909 ts_manager.cc:194] Registered new tserver with Master: 847579d6597946b29c17345d6f0de37d (127.12.78.3:46739)
I20260809 20:27:10.182515 27909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:41695
I20260809 20:27:10.190322 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:27:10.197305 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.191207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.12.78.62:40439)
0809 20:27:10.191449 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:10.191460 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:10.191901 (+ 441us) server_negotiation.cc:407] Connection header received
0809 20:27:10.192158 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:10.192162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:10.192294 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:10.192448 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:27:10.192902 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.193652 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.195105 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.195544 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.195689 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.196204 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:27:10.196741 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:27:10.196890 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:10.196977 (+ 87us) server_negotiation.cc:299] Negotiation successful
0809 20:27:10.197065 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 20:27:10.198248 27909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:27:10.206249 27909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:27:10.231827 28101 heartbeater.cc:503] Master 127.12.78.62:40439 was elected leader, sending a full tablet report...
I20260809 20:27:10.242707 28396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.219567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.12.78.1:46619)
0809 20:27:10.232472 (+ 12905us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:10.232484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:10.233865 (+ 1381us) server_negotiation.cc:407] Connection header received
0809 20:27:10.234107 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:10.234117 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:10.234432 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:10.234679 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:10.235605 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.237443 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.240054 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.241422 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.241584 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.241838 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:10.242162 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 20:27:10.242343 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12663,"thread_start_us":12938,"threads_started":1}
I20260809 20:27:10.243636 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.219011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:46619 (local address 127.0.0.1:40518)
0809 20:27:10.233727 (+ 14716us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:10.233744 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:10.233907 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:10.234712 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:10.234724 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:10.234778 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:10.235332 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:10.235349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.237668 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.237675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:10.239871 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:10.239885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.241685 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.241692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:10.241905 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.241947 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 20:27:10.242099 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 20:27:10.243223 (+ 1124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14583,"mutex_wait_us":26,"thread_start_us":13054,"threads_started":1}
I20260809 20:27:10.244752 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.218830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.12.78.2:38375)
0809 20:27:10.231848 (+ 13018us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:10.231862 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:10.231899 (+ 37us) server_negotiation.cc:407] Connection header received
0809 20:27:10.234733 (+ 2834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:10.234744 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:10.234990 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:10.235215 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:10.236780 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.238461 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.240975 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.242641 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.242783 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.244040 (+ 1257us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:10.244146 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 20:27:10.244345 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12824,"thread_start_us":13434,"threads_started":1}
I20260809 20:27:10.244913 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.218671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:38375 (local address 127.0.0.1:32944)
0809 20:27:10.231620 (+ 12949us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:10.231671 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:10.231862 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:10.235202 (+ 3340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:10.235208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:10.235240 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:10.236596 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:10.236611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.238644 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.238650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:10.240785 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:10.240802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.243199 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.243206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:10.243421 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.243464 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 20:27:10.243567 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 20:27:10.244467 (+ 900us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12645,"thread_start_us":14752,"threads_started":1}
I20260809 20:27:10.245482 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.232967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:46739 (local address 127.0.0.1:45724)
0809 20:27:10.236091 (+ 3124us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:10.236113 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:10.236821 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:10.237468 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:10.237472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:10.237497 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:10.237981 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:10.237992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.239869 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.239874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:10.242128 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:10.242144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.244247 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.244295 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:10.244464 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.244496 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 20:27:10.244626 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 20:27:10.244767 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2946,"mutex_wait_us":2,"thread_start_us":2772,"threads_started":1}
I20260809 20:27:10.245579 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.233214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.12.78.3:46739)
0809 20:27:10.236446 (+ 3232us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:10.236455 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:10.236492 (+ 37us) server_negotiation.cc:407] Connection header received
0809 20:27:10.237014 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:10.237024 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:10.237265 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:10.237477 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:10.238179 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.239703 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.242353 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.243955 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.244114 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.244349 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:10.245015 (+ 666us) server_negotiation.cc:299] Negotiation successful
0809 20:27:10.245219 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3052,"thread_start_us":2891,"threads_started":1}
I20260809 20:27:10.247897 28036 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9e5127e28d4711bfa3435476108c73 (DEFAULT_TABLE table=test-table [id=10630eb2df5e461ca9ff6896d0c28db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:27:10.249210 28036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9e5127e28d4711bfa3435476108c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:10.249035 28175 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9e5127e28d4711bfa3435476108c73 (DEFAULT_TABLE table=test-table [id=10630eb2df5e461ca9ff6896d0c28db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:27:10.249135 28310 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9e5127e28d4711bfa3435476108c73 (DEFAULT_TABLE table=test-table [id=10630eb2df5e461ca9ff6896d0c28db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:27:10.250552 28175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9e5127e28d4711bfa3435476108c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:10.250550 28310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9e5127e28d4711bfa3435476108c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:10.263046 28399 tablet_bootstrap.cc:492] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec: Bootstrap starting.
I20260809 20:27:10.266506 28399 tablet_bootstrap.cc:654] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:10.267643 28399 log.cc:824] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:10.268802 28401 tablet_bootstrap.cc:492] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d: Bootstrap starting.
I20260809 20:27:10.269754 28400 tablet_bootstrap.cc:492] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49: Bootstrap starting.
I20260809 20:27:10.273751 28400 tablet_bootstrap.cc:654] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:10.273995 28399 tablet_bootstrap.cc:492] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec: No bootstrap required, opened a new log
I20260809 20:27:10.274230 28401 tablet_bootstrap.cc:654] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:10.274319 28399 ts_tablet_manager.cc:1397] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20260809 20:27:10.275126 28400 log.cc:824] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:10.275396 28401 log.cc:824] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:10.278703 28400 tablet_bootstrap.cc:492] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49: No bootstrap required, opened a new log
I20260809 20:27:10.278703 28401 tablet_bootstrap.cc:492] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d: No bootstrap required, opened a new log
I20260809 20:27:10.279023 28401 ts_tablet_manager.cc:1397] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260809 20:27:10.279023 28400 ts_tablet_manager.cc:1397] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260809 20:27:10.282658 28399 raft_consensus.cc:348] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:10.283178 28399 raft_consensus.cc:374] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:10.283275 28399 raft_consensus.cc:729] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8370621deb3d465d840273954440c8ec, State: Initialized, Role: FOLLOWER
I20260809 20:27:10.284075 28399 consensus_queue.cc:260] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:10.286556 28399 ts_tablet_manager.cc:1428] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec: Time spent starting tablet: real 0.012s user 0.007s sys 0.006s
I20260809 20:27:10.291471 28400 raft_consensus.cc:348] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:10.291561 28401 raft_consensus.cc:348] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:10.292033 28400 raft_consensus.cc:374] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:10.292032 28401 raft_consensus.cc:374] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:10.292161 28400 raft_consensus.cc:729] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549192f2a5dd4ef8b3e27b28d6a1de49, State: Initialized, Role: FOLLOWER
I20260809 20:27:10.292157 28401 raft_consensus.cc:729] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847579d6597946b29c17345d6f0de37d, State: Initialized, Role: FOLLOWER
I20260809 20:27:10.292922 28401 consensus_queue.cc:260] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:10.292922 28400 consensus_queue.cc:260] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:10.294166 28242 heartbeater.cc:503] Master 127.12.78.62:40439 was elected leader, sending a full tablet report...
I20260809 20:27:10.294723 28381 heartbeater.cc:503] Master 127.12.78.62:40439 was elected leader, sending a full tablet report...
I20260809 20:27:10.295156 28400 ts_tablet_manager.cc:1428] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260809 20:27:10.295480 28401 ts_tablet_manager.cc:1428] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260809 20:27:10.401077 28383 tablet.cc:2369] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:27:10.415599 28243 tablet.cc:2369] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:27:10.446652 28102 tablet.cc:2369] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:27:10.519850 28407 raft_consensus.cc:482] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:27:10.520201 28407 raft_consensus.cc:504] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:10.522243 28407 leader_election.cc:288] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 549192f2a5dd4ef8b3e27b28d6a1de49 (127.12.78.2:38375), 8370621deb3d465d840273954440c8ec (127.12.78.1:46619)
I20260809 20:27:10.529256 28396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.523170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:56973 (local address 127.12.78.1:46619)
0809 20:27:10.523426 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:10.523432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:10.523453 (+ 21us) server_negotiation.cc:407] Connection header received
0809 20:27:10.523532 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:10.523550 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:10.523639 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:10.523740 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:10.524224 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.525164 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.527155 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.528298 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.528403 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.528481 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:10.528933 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 20:27:10.529047 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 20:27:10.529403 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.522570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:46619 (local address 127.12.78.3:56973)
0809 20:27:10.522799 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:10.522822 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:10.522962 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:10.523789 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:10.523794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:10.523825 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:10.524042 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:10.524055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.525315 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.525320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:10.526973 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:10.526986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.528480 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.528487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:10.528743 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.528785 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 20:27:10.528907 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 20:27:10.529079 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":58}
I20260809 20:27:10.530330 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.522668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:38375 (local address 127.12.78.3:51909)
0809 20:27:10.523528 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:10.523544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:10.523668 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:10.524045 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:10.524048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:10.524067 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:10.524461 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:10.524473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.525637 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.525641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:10.527839 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:10.527855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.529678 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.529686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:10.529833 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.529856 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 20:27:10.529937 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 20:27:10.530029 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":575,"threads_started":1}
I20260809 20:27:10.530368 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.522843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:51909 (local address 127.12.78.2:38375)
0809 20:27:10.523079 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:10.523087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:10.523640 (+ 553us) server_negotiation.cc:407] Connection header received
0809 20:27:10.523801 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:10.523806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:10.523883 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:10.523986 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:10.524670 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.525467 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.528036 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.529356 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.529528 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.529639 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:10.530002 (+ 363us) server_negotiation.cc:299] Negotiation successful
0809 20:27:10.530168 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 20:27:10.530757 28056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9e5127e28d4711bfa3435476108c73" candidate_uuid: "847579d6597946b29c17345d6f0de37d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8370621deb3d465d840273954440c8ec" is_pre_election: true
I20260809 20:27:10.531401 28056 raft_consensus.cc:2454] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 847579d6597946b29c17345d6f0de37d in term 0.
I20260809 20:27:10.531490 28195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9e5127e28d4711bfa3435476108c73" candidate_uuid: "847579d6597946b29c17345d6f0de37d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" is_pre_election: true
I20260809 20:27:10.531991 28195 raft_consensus.cc:2454] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 847579d6597946b29c17345d6f0de37d in term 0.
I20260809 20:27:10.532243 28263 leader_election.cc:302] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8370621deb3d465d840273954440c8ec, 847579d6597946b29c17345d6f0de37d; no voters:
I20260809 20:27:10.532685 28407 raft_consensus.cc:2781] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:27:10.532884 28407 raft_consensus.cc:482] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:27:10.532992 28407 raft_consensus.cc:3037] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:10.535365 28407 raft_consensus.cc:504] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:10.536059 28407 leader_election.cc:288] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [CANDIDATE]: Term 1 election: Requested vote from peers 549192f2a5dd4ef8b3e27b28d6a1de49 (127.12.78.2:38375), 8370621deb3d465d840273954440c8ec (127.12.78.1:46619)
I20260809 20:27:10.536394 28195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9e5127e28d4711bfa3435476108c73" candidate_uuid: "847579d6597946b29c17345d6f0de37d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49"
I20260809 20:27:10.536631 28195 raft_consensus.cc:3037] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:10.536777 28056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9e5127e28d4711bfa3435476108c73" candidate_uuid: "847579d6597946b29c17345d6f0de37d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8370621deb3d465d840273954440c8ec"
I20260809 20:27:10.536998 28056 raft_consensus.cc:3037] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:10.540050 28195 raft_consensus.cc:2454] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 847579d6597946b29c17345d6f0de37d in term 1.
I20260809 20:27:10.540330 28056 raft_consensus.cc:2454] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 847579d6597946b29c17345d6f0de37d in term 1.
I20260809 20:27:10.540524 28262 leader_election.cc:302] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549192f2a5dd4ef8b3e27b28d6a1de49, 847579d6597946b29c17345d6f0de37d; no voters:
I20260809 20:27:10.540912 28407 raft_consensus.cc:2781] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:27:10.541342 28407 raft_consensus.cc:686] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [term 1 LEADER]: Becoming Leader. State: Replica: 847579d6597946b29c17345d6f0de37d, State: Running, Role: LEADER
I20260809 20:27:10.541913 28407 consensus_queue.cc:237] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } }
I20260809 20:27:10.546593 27909 catalog_manager.cc:5495] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d reported cstate change: term changed from 0 to 1, leader changed from <none> to 847579d6597946b29c17345d6f0de37d (127.12.78.3). New cstate: current_term: 1 leader_uuid: "847579d6597946b29c17345d6f0de37d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "847579d6597946b29c17345d6f0de37d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 46739 } health_report { overall_health: HEALTHY } } }
I20260809 20:27:10.600227 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.594707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.12.78.3:46739)
0809 20:27:10.594927 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:10.594933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:10.594955 (+ 22us) server_negotiation.cc:407] Connection header received
0809 20:27:10.595063 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:10.595066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:10.595156 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:10.595315 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:27:10.595704 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.596439 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.598060 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.598458 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.598597 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.599120 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:27:10.599643 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:27:10.599804 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:10.599864 (+ 60us) server_negotiation.cc:299] Negotiation successful
0809 20:27:10.599947 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 20:27:10.608588 28056 raft_consensus.cc:1261] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec [term 1 FOLLOWER]: Refusing update from remote peer 847579d6597946b29c17345d6f0de37d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:27:10.608680 28195 raft_consensus.cc:1261] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 1 FOLLOWER]: Refusing update from remote peer 847579d6597946b29c17345d6f0de37d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:27:10.609589 28407 consensus_queue.cc:1035] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [LEADER]: Connected to new peer: Peer: permanent_uuid: "549192f2a5dd4ef8b3e27b28d6a1de49" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 38375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:27:10.609895 28410 consensus_queue.cc:1035] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8370621deb3d465d840273954440c8ec" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:27:10.633428 28414 mvcc.cc:205] Tried to move back new op lower bound from 7316714416563281920 to 7316714416303673344. Current Snapshot: MvccSnapshot[applied={T|T < 7316714416563281920}]
I20260809 20:27:10.641980 27908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:27:10.642316 27908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:27:10.646143 27908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Sending DeleteTablet for 3 replicas of tablet 7b9e5127e28d4711bfa3435476108c73
I20260809 20:27:10.647449 28036 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9e5127e28d4711bfa3435476108c73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:27:10 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:40518
I20260809 20:27:10.647415 28175 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9e5127e28d4711bfa3435476108c73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:27:10 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:32944
I20260809 20:27:10.648433 28424 tablet_replica.cc:330] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec: stopping tablet replica
I20260809 20:27:10.648593 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 27946
I20260809 20:27:10.648618 28310 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9e5127e28d4711bfa3435476108c73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:27:10 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:45724
I20260809 20:27:10.649163 28424 raft_consensus.cc:2229] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:27:10.649840 28424 raft_consensus.cc:2258] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:27:10.650049 28426 tablet_replica.cc:330] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d: stopping tablet replica
I20260809 20:27:10.650769 28426 raft_consensus.cc:2229] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:27:10.651485 28426 raft_consensus.cc:2258] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:27:10.653016 28424 ts_tablet_manager.cc:1905] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:27:10.653708 28426 ts_tablet_manager.cc:1905] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:27:10.661257 28426 ts_tablet_manager.cc:1918] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:27:10.661443 28426 log.cc:1196] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/7b9e5127e28d4711bfa3435476108c73
I20260809 20:27:10.662046 28426 ts_tablet_manager.cc:1939] T 7b9e5127e28d4711bfa3435476108c73 P 847579d6597946b29c17345d6f0de37d: Deleting consensus metadata
I20260809 20:27:10.663553 28424 ts_tablet_manager.cc:1918] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:27:10.663829 28424 log.cc:1196] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/7b9e5127e28d4711bfa3435476108c73
I20260809 20:27:10.664114 27894 catalog_manager.cc:4841] TS 847579d6597946b29c17345d6f0de37d (127.12.78.3:46739): tablet 7b9e5127e28d4711bfa3435476108c73 (table test-table [id=10630eb2df5e461ca9ff6896d0c28db7]) successfully deleted
I20260809 20:27:10.664569 28424 ts_tablet_manager.cc:1939] T 7b9e5127e28d4711bfa3435476108c73 P 8370621deb3d465d840273954440c8ec: Deleting consensus metadata
I20260809 20:27:10.666445 27894 catalog_manager.cc:4841] TS 8370621deb3d465d840273954440c8ec (127.12.78.1:46619): tablet 7b9e5127e28d4711bfa3435476108c73 (table test-table [id=10630eb2df5e461ca9ff6896d0c28db7]) successfully deleted
I20260809 20:27:10.669190 28396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.649455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.12.78.1:46619)
0809 20:27:10.650072 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:10.650079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:10.650098 (+ 19us) server_negotiation.cc:407] Connection header received
0809 20:27:10.650177 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:10.650181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:10.650280 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:10.650383 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:10.653049 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.654138 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.656170 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.656897 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.657022 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.660902 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:10.660920 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:10.660943 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:10.661001 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:10.665654 (+ 4653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:10.666681 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:10.666691 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:10.666695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:10.666858 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:10.667395 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:10.667401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:10.667403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:10.667879 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:10.668186 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:10.668588 (+ 402us) server_negotiation.cc:299] Negotiation successful
0809 20:27:10.668820 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488}
I20260809 20:27:10.667925 28425 tablet_replica.cc:330] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49: stopping tablet replica
I20260809 20:27:10.670135 28096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:10.670154 28425 raft_consensus.cc:2229] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:27:10.670774 28425 raft_consensus.cc:2258] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:27:10.672991 28425 ts_tablet_manager.cc:1905] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:27:10.681433 28425 ts_tablet_manager.cc:1918] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:27:10.681641 28425 log.cc:1196] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/7b9e5127e28d4711bfa3435476108c73
I20260809 20:27:10.682273 28425 ts_tablet_manager.cc:1939] T 7b9e5127e28d4711bfa3435476108c73 P 549192f2a5dd4ef8b3e27b28d6a1de49: Deleting consensus metadata
I20260809 20:27:10.685869 27893 catalog_manager.cc:4841] TS 549192f2a5dd4ef8b3e27b28d6a1de49 (127.12.78.2:38375): tablet 7b9e5127e28d4711bfa3435476108c73 (table test-table [id=10630eb2df5e461ca9ff6896d0c28db7]) successfully deleted
I20260809 20:27:10.778120 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 27946
I20260809 20:27:10.805627 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 28105
I20260809 20:27:10.821141 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.806522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.12.78.2:38375)
0809 20:27:10.806755 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:10.806762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:10.806785 (+ 23us) server_negotiation.cc:407] Connection header received
0809 20:27:10.806950 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:10.806956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:10.807046 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:10.807165 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:10.808498 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.809626 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.810946 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.811596 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.811734 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.815236 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:10.815250 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:10.815266 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:10.815303 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:10.818155 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:10.819030 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:10.819038 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:10.819040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:10.819184 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:10.819582 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:10.819586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:10.819589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:10.819949 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:10.820175 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:10.820638 (+ 463us) server_negotiation.cc:299] Negotiation successful
0809 20:27:10.820811 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 20:27:10.821878 28237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:10.912230 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 28105
I20260809 20:27:10.932613 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 28245
I20260809 20:27:10.951081 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:10.933731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.12.78.3:46739)
0809 20:27:10.933896 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:10.933901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:10.933919 (+ 18us) server_negotiation.cc:407] Connection header received
0809 20:27:10.933973 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:10.933977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:10.934043 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:10.934134 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:10.936124 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.937290 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.939193 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:10.939644 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:10.939749 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:10.943462 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:10.943478 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:10.943494 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:10.943535 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:10.946955 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:10.948035 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:10.948044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:10.948047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:10.948268 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:10.948811 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:10.948816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:10.948819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:10.949334 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:10.949666 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:10.950498 (+ 832us) server_negotiation.cc:299] Negotiation successful
0809 20:27:10.950738 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 20:27:10.951902 28376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:11.073321 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 28245
I20260809 20:27:11.096830 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 27873
I20260809 20:27:11.098204 27939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:11.214591 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 27873
I20260809 20:27:11.239925 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40439
--webserver_interface=127.12.78.62
--webserver_port=44707
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:40439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:27:11.597725 28431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:11.598122 28431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:11.598282 28431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:11.605921 28431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:27:11.606077 28431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:11.606196 28431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:11.606304 28431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:27:11.606381 28431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:27:11.617158 28431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:40439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=44707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:11.618662 28431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:11.620054 28431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:11.632792 28436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:11.632910 28439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:11.632753 28437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:11.633450 28431 server_base.cc:1034] running on GCE node
I20260809 20:27:11.634497 28431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:11.636094 28431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:11.637316 28431 hybrid_clock.cc:648] HybridClock initialized: now 1786307231637261 us; error 44 us; skew 500 ppm
I20260809 20:27:11.637729 28431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:11 dist-test-slave-rnrw krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307231, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:11.645094 28431 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:27:11.646837 28431 webserver.cc:460] Webserver started at http://127.12.78.62:44707/ using document root <none> and password file <none>
I20260809 20:27:11.647599 28431 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:11.647740 28431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:11.653559 28431 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 20:27:11.656515 28446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:11.658214 28431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 20:27:11.658417 28431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "0fff14088a4e4dcdae437f3723410567"
format_stamp: "Formatted at 2026-08-09 20:27:05 on dist-test-slave-rnrw"
server_key: "ce18f090a1f299fb4aea233f59cef1ad"
server_key_iv: "bbd8c6dc22f6152b71638c6a7670b5ae"
server_key_version: "encryptionkey@0"
I20260809 20:27:11.659509 28431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:11.695771 28431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:11.701169 28431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:11.701511 28431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:11.723033 28431 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:40439
I20260809 20:27:11.723053 28500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:40439 every 8 connection(s)
I20260809 20:27:11.725041 28431 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:27:11.728713 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 28431
I20260809 20:27:11.729823 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:46619
--local_ip_for_outbound_sockets=127.12.78.1
--tserver_master_addrs=127.12.78.62:40439
--webserver_port=45049
--webserver_interface=127.12.78.1
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:27:11.735168 28501 sys_catalog.cc:263] Verifying existing consensus state
I20260809 20:27:11.740756 28501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Bootstrap starting.
I20260809 20:27:11.762157 28501 log.cc:824] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:11.779436 28501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 20:27:11.780036 28501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567: Bootstrap complete.
I20260809 20:27:11.790040 28501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } }
I20260809 20:27:11.791007 28501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fff14088a4e4dcdae437f3723410567, State: Initialized, Role: FOLLOWER
I20260809 20:27:11.791632 28501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } }
I20260809 20:27:11.791880 28501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:27:11.792032 28501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:27:11.792228 28501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 2 FOLLOWER]: Advancing to term 3
I20260809 20:27:11.796267 28501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } }
I20260809 20:27:11.796833 28501 leader_election.cc:302] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fff14088a4e4dcdae437f3723410567; no voters:
I20260809 20:27:11.797587 28501 leader_election.cc:288] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 20:27:11.797749 28507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 20:27:11.798890 28507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [term 3 LEADER]: Becoming Leader. State: Replica: 0fff14088a4e4dcdae437f3723410567, State: Running, Role: LEADER
I20260809 20:27:11.799456 28507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } }
I20260809 20:27:11.800001 28501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:27:11.809494 28509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fff14088a4e4dcdae437f3723410567. Latest consensus state: current_term: 3 leader_uuid: "0fff14088a4e4dcdae437f3723410567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } } }
I20260809 20:27:11.809928 28509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: This master's current role is: LEADER
I20260809 20:27:11.810735 28508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0fff14088a4e4dcdae437f3723410567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fff14088a4e4dcdae437f3723410567" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40439 } } }
I20260809 20:27:11.811069 28508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fff14088a4e4dcdae437f3723410567 [sys.catalog]: This master's current role is: LEADER
I20260809 20:27:11.818842 28520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:27:11.836277 28520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:27:11.838470 28520 catalog_manager.cc:1259] Loaded cluster ID: 08dc62be39534972a5ce404cd5d9f274
I20260809 20:27:11.838624 28520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 20:27:11.841897 28520 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f781add59ad google::LogMessage::Fail() at ??:0
@ 0x7f781add888a google::LogMessage::SendToLog() at ??:0
@ 0x7f781add54de google::LogMessage::Flush() at ??:0
@ 0x7f781add6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260809 20:27:11.857692 28526 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f783080c512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7830809976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7830891610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7830891439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f781da21345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f781da2491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f781da24799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f781d9ffeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7820cd96db start_thread at ??:0
@ 0x7f7819c6f61f clone at ??:0
W20260809 20:27:12.160748 28503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:12.161093 28503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:12.161273 28503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:12.168891 28503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:12.169050 28503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:12.169237 28503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:27:12.179301 28503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:46619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=45049
--tserver_master_addrs=127.12.78.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:12.180809 28503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:12.182235 28503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:12.196029 28545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:12.196115 28546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:12.196085 28548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:12.197228 28503 server_base.cc:1034] running on GCE node
I20260809 20:27:12.197872 28503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:12.198853 28503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:12.200070 28503 hybrid_clock.cc:648] HybridClock initialized: now 1786307232200029 us; error 37 us; skew 500 ppm
I20260809 20:27:12.200381 28503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:12 dist-test-slave-rnrw krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307232, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:12.203724 28503 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:27:12.205317 28503 webserver.cc:460] Webserver started at http://127.12.78.1:45049/ using document root <none> and password file <none>
I20260809 20:27:12.206060 28503 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:12.206203 28503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:12.211980 28503 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 20:27:12.215255 28555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:12.216799 28503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:27:12.216977 28503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "8370621deb3d465d840273954440c8ec"
format_stamp: "Formatted at 2026-08-09 20:27:05 on dist-test-slave-rnrw"
server_key: "70b9dbf59576ea10709938359335128d"
server_key_iv: "b56599bbd1a9c7c1b5940d491e1f316e"
server_key_version: "encryptionkey@0"
I20260809 20:27:12.218081 28503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:12.235072 28503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:12.239449 28503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:12.239810 28503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:12.240881 28503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:12.242262 28503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:12.242381 28503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:12.242513 28503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:12.242645 28503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:12.278630 28503 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:46619
I20260809 20:27:12.278748 28672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:46619 every 8 connection(s)
I20260809 20:27:12.280227 28503 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:27:12.285810 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:12.283632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:48705)
0809 20:27:12.284457 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:12.284787 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":351,"threads_started":1}
I20260809 20:27:12.286453 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 28503
W20260809 20:27:12.286743 28673 heartbeater.cc:644] Failed to heartbeat to 127.12.78.62:40439 (0 consecutive failures): Network error: Failed to ping master at 127.12.78.62:40439: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
I20260809 20:27:12.287284 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:38375
--local_ip_for_outbound_sockets=127.12.78.2
--tserver_master_addrs=127.12.78.62:40439
--webserver_port=42611
--webserver_interface=127.12.78.2
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:27:12.288009 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:12.287405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:33987)
0809 20:27:12.287713 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:12.287830 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260809 20:27:12.289475 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:12.288986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:45847)
0809 20:27:12.289182 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:12.289297 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 20:27:12.289870 28673 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 20:27:12.642783 28676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:12.643069 28676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:12.643167 28676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:12.650635 28676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:12.650756 28676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:12.650874 28676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:27:12.661221 28676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:38375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=42611
--tserver_master_addrs=127.12.78.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:12.662999 28676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:12.664366 28676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:12.678359 28685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:12.678399 28682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:12.678318 28681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:12.679132 28676 server_base.cc:1034] running on GCE node
I20260809 20:27:12.679765 28676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:12.680830 28676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:12.682070 28676 hybrid_clock.cc:648] HybridClock initialized: now 1786307232682033 us; error 57 us; skew 500 ppm
I20260809 20:27:12.682399 28676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:12 dist-test-slave-rnrw krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307232, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:12.685672 28676 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:27:12.687275 28676 webserver.cc:460] Webserver started at http://127.12.78.2:42611/ using document root <none> and password file <none>
I20260809 20:27:12.687997 28676 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:12.688143 28676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:12.693626 28676 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:27:12.696377 28692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:12.697959 28676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 20:27:12.698158 28676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "549192f2a5dd4ef8b3e27b28d6a1de49"
format_stamp: "Formatted at 2026-08-09 20:27:06 on dist-test-slave-rnrw"
server_key: "1eb5f9eca6bef5b14e3278062c431c81"
server_key_iv: "6fd96d274bbbc48ac1a3be001fbbe92b"
server_key_version: "encryptionkey@0"
I20260809 20:27:12.699252 28676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:12.729761 28676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:12.734150 28676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:12.734457 28676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:12.735508 28676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:12.737036 28676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:12.737161 28676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:12.737291 28676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:12.737385 28676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:12.772254 28676 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:38375
I20260809 20:27:12.772344 28811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:38375 every 8 connection(s)
I20260809 20:27:12.773806 28676 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:27:12.777113 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 28676
I20260809 20:27:12.778048 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:46739
--local_ip_for_outbound_sockets=127.12.78.3
--tserver_master_addrs=127.12.78.62:40439
--webserver_port=38523
--webserver_interface=127.12.78.3
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:27:12.780455 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:12.777247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:59775)
0809 20:27:12.778792 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:12.779086 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":490,"threads_started":1}
W20260809 20:27:12.781391 28812 heartbeater.cc:644] Failed to heartbeat to 127.12.78.62:40439 (0 consecutive failures): Network error: Failed to ping master at 127.12.78.62:40439: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
I20260809 20:27:12.782630 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:12.782090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:34779)
0809 20:27:12.782373 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:12.782467 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260809 20:27:12.784307 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:12.783773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:56885)
0809 20:27:12.784019 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:12.784134 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260809 20:27:12.784684 28812 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 20:27:13.124230 28815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:13.124528 28815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:13.124660 28815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:13.131916 28815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:13.132036 28815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:13.132151 28815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:27:13.141981 28815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:46739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=38523
--tserver_master_addrs=127.12.78.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:13.143283 28815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:13.144632 28815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:13.158114 28820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:13.158211 28821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:13.158214 28823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:13.159564 28815 server_base.cc:1034] running on GCE node
I20260809 20:27:13.160243 28815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:13.161304 28815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:13.162525 28815 hybrid_clock.cc:648] HybridClock initialized: now 1786307233162479 us; error 33 us; skew 500 ppm
I20260809 20:27:13.162919 28815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:13 dist-test-slave-rnrw krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307233, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:13.166308 28815 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:27:13.167959 28815 webserver.cc:460] Webserver started at http://127.12.78.3:38523/ using document root <none> and password file <none>
I20260809 20:27:13.168776 28815 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:13.168926 28815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:13.174602 28815 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 20:27:13.177698 28830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:13.179210 28815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 20:27:13.179397 28815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "847579d6597946b29c17345d6f0de37d"
format_stamp: "Formatted at 2026-08-09 20:27:07 on dist-test-slave-rnrw"
server_key: "15737e438a8ba4c53bd37e7833688652"
server_key_iv: "8d12daff831c455eeed431d778f9f018"
server_key_version: "encryptionkey@0"
I20260809 20:27:13.180425 28815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:13.217954 28815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:13.223291 28815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:13.223697 28815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:13.225028 28815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:13.226581 28815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:13.226727 28815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:13.226859 28815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:13.226970 28815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:13.265277 28815 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:46739
I20260809 20:27:13.265349 28946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:46739 every 8 connection(s)
I20260809 20:27:13.267036 28815 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:27:13.268908 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 28815
I20260809 20:27:13.273029 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:13.270789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:53917)
0809 20:27:13.271712 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:13.271933 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":364,"threads_started":1}
W20260809 20:27:13.273919 28947 heartbeater.cc:644] Failed to heartbeat to 127.12.78.62:40439 (0 consecutive failures): Network error: Failed to ping master at 127.12.78.62:40439: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
I20260809 20:27:13.274868 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:13.274495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:60167)
0809 20:27:13.274673 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:13.274747 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 20:27:13.275993 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:13.275597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:34541)
0809 20:27:13.275775 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:13.275864 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 20:27:13.276253 28947 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 20:27:13.291438 28951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:13.290461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:37315)
0809 20:27:13.291016 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:13.291163 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":270,"threads_started":1}
I20260809 20:27:13.786185 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:13.785277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:44579)
0809 20:27:13.785808 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:13.785915 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":240,"threads_started":1}
I20260809 20:27:14.277969 28953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:14.276928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:47079)
0809 20:27:14.277449 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:14.277619 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":264,"threads_started":1}
I20260809 20:27:14.293303 28954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:14.292440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:43895)
0809 20:27:14.293007 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:14.293114 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":310,"threads_started":1}
I20260809 20:27:14.788511 28955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:14.787263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:58907)
0809 20:27:14.788236 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:14.788333 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":653,"threads_started":1}
I20260809 20:27:15.280064 28956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:15.279230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:58471)
0809 20:27:15.279761 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:15.279863 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":261,"threads_started":1}
I20260809 20:27:15.295224 28957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:15.294322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:40925)
0809 20:27:15.294853 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:15.294939 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":308,"threads_started":1}
I20260809 20:27:15.790179 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:15.789398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:33391)
0809 20:27:15.789891 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:15.789992 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":259,"threads_started":1}
I20260809 20:27:16.281785 28959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:16.281102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:51611)
0809 20:27:16.281534 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:16.281616 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":224,"threads_started":1}
I20260809 20:27:16.297163 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:16.296209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:39209)
0809 20:27:16.296831 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:16.296949 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":227,"threads_started":1}
I20260809 20:27:16.791981 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:16.791193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:39877)
0809 20:27:16.791703 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:16.791802 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":254,"threads_started":1}
I20260809 20:27:17.283753 28962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:17.282876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:57355)
0809 20:27:17.283467 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:17.283572 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":318,"threads_started":1}
I20260809 20:27:17.298964 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:17.298145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:44389)
0809 20:27:17.298688 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:17.298779 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":279,"threads_started":1}
I20260809 20:27:17.793767 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:17.793056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:54031)
0809 20:27:17.793513 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:17.793593 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":250,"threads_started":1}
I20260809 20:27:18.285528 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:18.284782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:36713)
0809 20:27:18.285282 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:18.285361 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":273,"threads_started":1}
I20260809 20:27:18.300853 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:18.299995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:43569)
0809 20:27:18.300505 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:18.300677 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":269,"threads_started":1}
I20260809 20:27:18.795375 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:18.794617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:56009)
0809 20:27:18.795105 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:18.795206 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":249,"threads_started":1}
I20260809 20:27:19.287319 28968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:19.286492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:38067)
0809 20:27:19.287021 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:19.287155 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":280,"threads_started":1}
I20260809 20:27:19.302923 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:19.301912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:40707)
0809 20:27:19.302599 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:19.302693 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":287,"threads_started":1}
I20260809 20:27:19.797238 28970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:19.796403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:50875)
0809 20:27:19.796942 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:19.797031 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":270,"threads_started":1}
I20260809 20:27:20.289294 28971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:20.288330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:45851)
0809 20:27:20.288897 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:20.288994 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":283,"threads_started":1}
I20260809 20:27:20.304879 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:20.304060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:33961)
0809 20:27:20.304617 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:20.304708 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":237,"threads_started":1}
I20260809 20:27:20.798938 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:20.798209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:49501)
0809 20:27:20.798685 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:20.798774 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":237,"threads_started":1}
I20260809 20:27:21.291106 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:21.290332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:51283)
0809 20:27:21.290818 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:21.290909 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
I20260809 20:27:21.306826 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:21.305961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:35141)
0809 20:27:21.306533 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:21.306640 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":291,"threads_started":1}
I20260809 20:27:21.801268 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:21.799994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:46095)
0809 20:27:21.800965 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:21.801091 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":709,"threads_started":1}
I20260809 20:27:22.293043 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:22.292151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:32993)
0809 20:27:22.292743 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:22.292865 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":261,"threads_started":1}
I20260809 20:27:22.308825 28979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:22.308001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:47179)
0809 20:27:22.308470 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:22.308651 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":253,"threads_started":1}
I20260809 20:27:22.803067 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:22.802228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:56583)
0809 20:27:22.802804 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:22.802890 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":355,"threads_started":1}
I20260809 20:27:23.294811 28981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:23.294004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:46939)
0809 20:27:23.294511 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:23.294631 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":294,"threads_started":1}
I20260809 20:27:23.310676 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:23.309898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:37279)
0809 20:27:23.310376 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:23.310478 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":242,"threads_started":1}
I20260809 20:27:23.804852 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:23.803990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:37963)
0809 20:27:23.804499 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:23.804649 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":221,"threads_started":1}
I20260809 20:27:24.296684 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:24.295762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:47947)
0809 20:27:24.296252 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:24.296360 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":243,"threads_started":1}
I20260809 20:27:24.312536 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:24.311734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:54065)
0809 20:27:24.312287 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:24.312373 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":293,"threads_started":1}
I20260809 20:27:24.806577 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:24.805867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:38949)
0809 20:27:24.806330 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:24.806417 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":224,"threads_started":1}
I20260809 20:27:25.298552 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:25.297737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:55947)
0809 20:27:25.298282 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:25.298382 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":293,"threads_started":1}
I20260809 20:27:25.314774 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:25.313900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:57011)
0809 20:27:25.314469 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:25.314576 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":271,"threads_started":1}
I20260809 20:27:25.808324 28989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:25.807536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:39809)
0809 20:27:25.808031 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:25.808132 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":235,"threads_started":1}
I20260809 20:27:26.300331 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:26.299549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:57611)
0809 20:27:26.300037 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:26.300138 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":238,"threads_started":1}
I20260809 20:27:26.316926 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:26.315928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:58903)
0809 20:27:26.316496 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:26.316671 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":316,"threads_started":1}
I20260809 20:27:26.810036 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:26.809295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:55719)
0809 20:27:26.809753 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:26.809838 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":240,"threads_started":1}
I20260809 20:27:27.302120 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:27.301367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:50775)
0809 20:27:27.301831 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:27.301919 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":234,"threads_started":1}
I20260809 20:27:27.319324 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:27.318282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:57507)
0809 20:27:27.318933 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:27.319058 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":284,"threads_started":1}
I20260809 20:27:27.812037 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:27.811197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:33705)
0809 20:27:27.811755 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:27.811851 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":302,"threads_started":1}
I20260809 20:27:28.283588 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 28503
I20260809 20:27:28.304376 28997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:28.303543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.3:37199)
0809 20:27:28.304047 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:28.304185 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":258,"threads_started":1}
I20260809 20:27:28.321609 28998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:28.320425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.1:59585)
0809 20:27:28.321191 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:28.321371 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":438,"threads_started":1}
I20260809 20:27:28.813925 28999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:28.813119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40439 (local address 127.12.78.2:51113)
0809 20:27:28.813659 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:28.813760 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:40439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":269,"threads_started":1}
I20260809 20:27:28.955873 28996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:28.285540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.12.78.1:46619)
0809 20:27:28.286230 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:28.286245 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:28.937503 (+651258us) server_negotiation.cc:407] Connection header received
0809 20:27:28.939360 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:28.939371 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:28.939881 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:28.940252 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:28.942024 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:28.943373 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:28.944868 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:28.945585 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:28.945739 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:28.948945 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:28.948955 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:28.948973 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:28.949025 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:28.952163 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:28.953048 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:28.953055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:28.953062 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:28.953240 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:28.953683 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:28.953687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:28.953689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:28.954277 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:28.954609 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:28.955084 (+ 475us) server_negotiation.cc:299] Negotiation successful
0809 20:27:28.955557 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":358,"threads_started":1}
I20260809 20:27:28.957607 28668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:29.045076 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 28503
I20260809 20:27:29.066334 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 28676
I20260809 20:27:29.085135 29001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:29.067511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.12.78.2:38375)
0809 20:27:29.068146 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:29.068159 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:29.068199 (+ 40us) server_negotiation.cc:407] Connection header received
0809 20:27:29.070033 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:29.070043 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:29.070475 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:29.070834 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:29.072373 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:29.073655 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:29.074964 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:29.075650 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:29.075846 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:29.078659 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:29.078667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:29.078681 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:29.078718 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:29.081667 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:29.082366 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:29.082373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:29.082380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:29.082568 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:29.082992 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:29.082998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:29.083001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:29.083542 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:29.083852 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:29.084396 (+ 544us) server_negotiation.cc:299] Negotiation successful
0809 20:27:29.084839 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":304,"threads_started":1}
I20260809 20:27:29.086593 28807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:29.171864 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 28676
I20260809 20:27:29.191974 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 28815
I20260809 20:27:29.210637 29003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:29.192996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.12.78.3:46739)
0809 20:27:29.193535 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:29.193549 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:29.193588 (+ 39us) server_negotiation.cc:407] Connection header received
0809 20:27:29.195134 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:29.195145 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:29.195567 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:29.195931 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:29.197489 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:29.198711 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:29.200020 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:29.200782 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:29.200981 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:29.203814 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:29.203822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:29.203838 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:29.203880 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:29.207054 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:29.207859 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:29.207867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:29.207873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:29.208054 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:29.208466 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:29.208471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:29.208475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:29.209075 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:29.209415 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:29.209886 (+ 471us) server_negotiation.cc:299] Negotiation successful
0809 20:27:29.210351 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":252,"threads_started":1}
I20260809 20:27:29.212141 28942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:29.296682 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 28815
2026-08-09T20:27:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25794 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:27:29 dist-test-slave-rnrw krb5kdc[29007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:27:29 dist-test-slave-rnrw krb5kdc[29007](info): set up 2 sockets
Aug 09 20:27:29 dist-test-slave-rnrw krb5kdc[29007](info): commencing operation
krb5kdc: starting...
W20260809 20:27:30.445521 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.011s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:27:30 dist-test-slave-rnrw krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:27:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:27:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:27:30.733683 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:42277
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:42277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:27:31.082351 29020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:31.082661 29020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:31.082837 29020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:31.090188 29020 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 20:27:31.090338 29020 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 20:27:31.090411 29020 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 20:27:31.090515 29020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:27:31.090624 29020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:31.090749 29020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:31.090818 29020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:27:31.090876 29020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:27:31.101130 29020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:42277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:42277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:31.102411 29020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:31.103718 29020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:31.115990 29026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:31.116113 29025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:31.116070 29028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:31.116901 29020 server_base.cc:1034] running on GCE node
I20260809 20:27:31.117719 29020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:31.119210 29020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:31.120467 29020 hybrid_clock.cc:648] HybridClock initialized: now 1786307251120354 us; error 95 us; skew 500 ppm
I20260809 20:27:31.120855 29020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:31 dist-test-slave-rnrw krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307251, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:31.127835 29020 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:27:31.129539 29020 webserver.cc:460] Webserver started at http://127.12.78.62:34291/ using document root <none> and password file <none>
I20260809 20:27:31.130298 29020 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:31.130435 29020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:31.130776 29020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:27:31.133901 29020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "a9041503701d4e079f66889a0a249e58"
format_stamp: "Formatted at 2026-08-09 20:27:31 on dist-test-slave-rnrw"
server_key: "99391ddb2d1e8d28ff12284d6f8f0fd2"
server_key_iv: "6a1ac375fc5c777e617237ffcfd84307"
server_key_version: "encryptionkey@0"
I20260809 20:27:31.134714 29020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "a9041503701d4e079f66889a0a249e58"
format_stamp: "Formatted at 2026-08-09 20:27:31 on dist-test-slave-rnrw"
server_key: "99391ddb2d1e8d28ff12284d6f8f0fd2"
server_key_iv: "6a1ac375fc5c777e617237ffcfd84307"
server_key_version: "encryptionkey@0"
I20260809 20:27:31.140013 29020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 20:27:31.143628 29035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:31.145190 29020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 20:27:31.145413 29020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "a9041503701d4e079f66889a0a249e58"
format_stamp: "Formatted at 2026-08-09 20:27:31 on dist-test-slave-rnrw"
server_key: "99391ddb2d1e8d28ff12284d6f8f0fd2"
server_key_iv: "6a1ac375fc5c777e617237ffcfd84307"
server_key_version: "encryptionkey@0"
I20260809 20:27:31.145685 29020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:31.176833 29020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:31.182057 29020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:31.182394 29020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:31.203985 29020 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:42277
I20260809 20:27:31.204017 29089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:42277 every 8 connection(s)
I20260809 20:27:31.205705 29020 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:27:31.209616 29090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:31.211796 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 29020
I20260809 20:27:31.212044 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:27:31.212929 12600 external_mini_cluster.cc:1428] Setting key b31337f10734a702d538026745a525f8
I20260809 20:27:31.222335 29090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Bootstrap starting.
I20260809 20:27:31.225636 29090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:31.226847 29090 log.cc:824] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Log is configured to *not* fsync() on all Append() calls
Aug 09 20:27:31 dist-test-slave-rnrw krb5kdc[29007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:31.230571 29090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: No bootstrap required, opened a new log
I20260809 20:27:31.239261 29090 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } }
I20260809 20:27:31.239696 29090 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:31.239845 29090 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9041503701d4e079f66889a0a249e58, State: Initialized, Role: FOLLOWER
I20260809 20:27:31.239976 29093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:31.215366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40134 (local address 127.12.78.62:42277)
0809 20:27:31.216275 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:31.216292 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:31.216340 (+ 48us) server_negotiation.cc:407] Connection header received
0809 20:27:31.218274 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:31.218301 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:31.218737 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:31.219229 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:31.221085 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:31.222641 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:31.224200 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:31.225053 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:31.225275 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:31.230976 (+ 5701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:31.230986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:31.231006 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:31.231058 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:31.234899 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:31.235899 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:31.235907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:31.235919 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:31.236138 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:31.236667 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:31.236674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:31.236681 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:31.237325 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:31.237680 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:31.238242 (+ 562us) server_negotiation.cc:299] Negotiation successful
0809 20:27:31.238729 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":399,"threads_started":1}
I20260809 20:27:31.240639 29090 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } }
I20260809 20:27:31.240984 29090 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:27:31.241149 29090 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:27:31.241339 29090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:31.243827 29090 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } }
I20260809 20:27:31.244397 29090 leader_election.cc:302] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9041503701d4e079f66889a0a249e58; no voters:
I20260809 20:27:31.245143 29090 leader_election.cc:288] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:27:31.245311 29095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:27:31.246086 29095 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 1 LEADER]: Becoming Leader. State: Replica: a9041503701d4e079f66889a0a249e58, State: Running, Role: LEADER
I20260809 20:27:31.246730 29095 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } }
I20260809 20:27:31.247489 29090 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:27:31.250052 29096 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9041503701d4e079f66889a0a249e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } } }
I20260809 20:27:31.250483 29096 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: This master's current role is: LEADER
I20260809 20:27:31.250975 29097 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9041503701d4e079f66889a0a249e58. Latest consensus state: current_term: 1 leader_uuid: "a9041503701d4e079f66889a0a249e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } } }
I20260809 20:27:31.251291 29097 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: This master's current role is: LEADER
I20260809 20:27:31.253691 29101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:27:31.261294 29101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:27:31.271828 29101 catalog_manager.cc:1347] Generated new cluster ID: f0b8b1bd09644b7aa8dc00c67995a08d
I20260809 20:27:31.271965 29101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:27:31.281738 29101 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:27:31.283735 29101 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:27:31.303499 29101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Generated new TSK 0
I20260809 20:27:31.304663 29101 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:27:31.380781 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42277
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:27:31.733107 29117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:31.733467 29117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:31.733650 29117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:31.741307 29117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:31.741458 29117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:31.741640 29117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:27:31.752024 29117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:31.753388 29117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:31.754765 29117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:31.768070 29122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:31.768036 29123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:31.768110 29125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:31.768954 29117 server_base.cc:1034] running on GCE node
I20260809 20:27:31.769542 29117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:31.770609 29117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:31.771816 29117 hybrid_clock.cc:648] HybridClock initialized: now 1786307251771774 us; error 28 us; skew 500 ppm
I20260809 20:27:31.772186 29117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:31 dist-test-slave-rnrw krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307251, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:31.775558 29117 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:27:31.777179 29117 webserver.cc:460] Webserver started at http://127.12.78.1:41487/ using document root <none> and password file <none>
I20260809 20:27:31.777901 29117 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:31.778059 29117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:31.778384 29117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:27:31.781548 29117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "670837917ff8439386f4315d00fc8004"
format_stamp: "Formatted at 2026-08-09 20:27:31 on dist-test-slave-rnrw"
server_key: "40c9e00144d1e680b42d1c26fbb1db1f"
server_key_iv: "5dfc7bf660a38693fbd3ba48015167e6"
server_key_version: "encryptionkey@0"
I20260809 20:27:31.782341 29117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "670837917ff8439386f4315d00fc8004"
format_stamp: "Formatted at 2026-08-09 20:27:31 on dist-test-slave-rnrw"
server_key: "40c9e00144d1e680b42d1c26fbb1db1f"
server_key_iv: "5dfc7bf660a38693fbd3ba48015167e6"
server_key_version: "encryptionkey@0"
I20260809 20:27:31.787528 29117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:27:31.791045 29132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:31.792385 29117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 20:27:31.792711 29117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "670837917ff8439386f4315d00fc8004"
format_stamp: "Formatted at 2026-08-09 20:27:31 on dist-test-slave-rnrw"
server_key: "40c9e00144d1e680b42d1c26fbb1db1f"
server_key_iv: "5dfc7bf660a38693fbd3ba48015167e6"
server_key_version: "encryptionkey@0"
I20260809 20:27:31.792909 29117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:31.811930 29117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:31.816157 29117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:31.816478 29117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:31.817526 29117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:31.818923 29117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:31.818998 29117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:31.819177 29117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:31.819242 29117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:31.858955 29117 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:34943
I20260809 20:27:31.858999 29246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:34943 every 8 connection(s)
I20260809 20:27:31.860250 29117 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:27:31.862133 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 29117
I20260809 20:27:31.862393 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:27:31.863328 12600 external_mini_cluster.cc:1428] Setting key 6ae3ca2b6efbccaa9e07360cd19bf135
Aug 09 20:27:31 dist-test-slave-rnrw krb5kdc[29007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307251, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:31.880815 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:31.862974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:52117 (local address 127.12.78.62:42277)
0809 20:27:31.863484 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:31.863490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:31.864801 (+ 1311us) server_negotiation.cc:407] Connection header received
0809 20:27:31.866946 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:31.866951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:31.867040 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:31.867163 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:31.868767 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:31.869504 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:31.871048 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:31.871453 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:31.871614 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:31.875099 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:31.875104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:31.875107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:31.875147 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:31.877746 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:31.878472 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:31.878478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:31.878486 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:31.878608 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:31.879035 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:31.879040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:31.879043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:31.879319 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:31.879475 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:31.880185 (+ 710us) server_negotiation.cc:299] Negotiation successful
0809 20:27:31.880415 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":278,"threads_started":1}
I20260809 20:27:31.881779 29250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:31.863695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:52117)
0809 20:27:31.864615 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:31.864667 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:31.866657 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:31.867410 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:31.867418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:31.867973 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:27:31.868604 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:31.868625 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:31.869676 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:31.869680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:31.870888 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:31.870901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:31.871720 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:31.871730 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:31.871962 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:31.872690 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:27:31.872717 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:27:31.874871 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:27:31.877890 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:31.877899 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:31.877917 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:31.878343 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:31.878715 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:31.878720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:31.878724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:31.878915 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:31.879448 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:27:31.879456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:27:31.879804 (+ 348us) client_negotiation.cc:769] Sending connection context
0809 20:27:31.880071 (+ 267us) client_negotiation.cc:240] Negotiation successful
0809 20:27:31.880637 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":364,"threads_started":1}
I20260809 20:27:31.883422 29247 heartbeater.cc:347] Connected to a master server at 127.12.78.62:42277
I20260809 20:27:31.883726 29247 heartbeater.cc:464] Registering TS with master...
I20260809 20:27:31.884447 29247 heartbeater.cc:511] Master 127.12.78.62:42277 requested a full tablet report, sending...
I20260809 20:27:31.886885 29054 ts_manager.cc:194] Registered new tserver with Master: 670837917ff8439386f4315d00fc8004 (127.12.78.1:34943)
I20260809 20:27:31.889276 29054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:52117
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:27:31.927738 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42277
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:27:32.273170 29253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:32.273568 29253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:32.273758 29253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:32.281250 29253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:32.281374 29253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:32.281558 29253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:27:32.291703 29253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:32.293116 29253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:32.294585 29253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:32.308349 29259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:32.308418 29258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:32.308944 29261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:32.309171 29253 server_base.cc:1034] running on GCE node
I20260809 20:27:32.309875 29253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:32.310904 29253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:32.312119 29253 hybrid_clock.cc:648] HybridClock initialized: now 1786307252312051 us; error 64 us; skew 500 ppm
I20260809 20:27:32.312464 29253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:32 dist-test-slave-rnrw krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307252, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:32.315812 29253 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:27:32.317493 29253 webserver.cc:460] Webserver started at http://127.12.78.2:35361/ using document root <none> and password file <none>
I20260809 20:27:32.318228 29253 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:32.318392 29253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:32.318724 29253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:27:32.321806 29253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "468cc626e28c473ebaf71388ed6b1e69"
format_stamp: "Formatted at 2026-08-09 20:27:32 on dist-test-slave-rnrw"
server_key: "1c607ceec687b50da5002c8943be9c7e"
server_key_iv: "3b827a162327df7e574ca83e0a675c19"
server_key_version: "encryptionkey@0"
I20260809 20:27:32.322614 29253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "468cc626e28c473ebaf71388ed6b1e69"
format_stamp: "Formatted at 2026-08-09 20:27:32 on dist-test-slave-rnrw"
server_key: "1c607ceec687b50da5002c8943be9c7e"
server_key_iv: "3b827a162327df7e574ca83e0a675c19"
server_key_version: "encryptionkey@0"
I20260809 20:27:32.328099 29253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:27:32.331815 29269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:32.333209 29253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:27:32.333400 29253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "468cc626e28c473ebaf71388ed6b1e69"
format_stamp: "Formatted at 2026-08-09 20:27:32 on dist-test-slave-rnrw"
server_key: "1c607ceec687b50da5002c8943be9c7e"
server_key_iv: "3b827a162327df7e574ca83e0a675c19"
server_key_version: "encryptionkey@0"
I20260809 20:27:32.333604 29253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:32.347883 29253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:32.352262 29253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:32.352623 29253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:32.353616 29253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:32.355038 29253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:32.355141 29253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:32.355290 29253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:32.355366 29253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:32.389261 29253 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:39647
I20260809 20:27:32.389312 29385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:39647 every 8 connection(s)
I20260809 20:27:32.390583 29253 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:27:32.396032 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 29253
I20260809 20:27:32.396278 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:27:32.396924 12600 external_mini_cluster.cc:1428] Setting key 364a56c4ecad9f278f2a06a36994b654
Aug 09 20:27:32 dist-test-slave-rnrw krb5kdc[29007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307252, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:32.412896 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:32.393228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:45825 (local address 127.12.78.62:42277)
0809 20:27:32.393650 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:32.393655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:32.394661 (+ 1006us) server_negotiation.cc:407] Connection header received
0809 20:27:32.397123 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:32.397127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:32.397206 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:32.397367 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:32.399404 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:32.400134 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:32.401773 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:32.402217 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:32.402352 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:32.406421 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:32.406427 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:32.406430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:32.406465 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:32.409538 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:32.410384 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:32.410389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:32.410391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:32.410567 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:32.411031 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:32.411042 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:32.411045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:32.411311 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:32.411475 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:32.412198 (+ 723us) server_negotiation.cc:299] Negotiation successful
0809 20:27:32.412466 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":199,"threads_started":1}
I20260809 20:27:32.413939 29389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:32.393747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:45825)
0809 20:27:32.394473 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:32.394510 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:32.396805 (+ 2295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:32.397677 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:32.397689 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:32.398421 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:27:32.399226 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:32.399247 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:32.400304 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:32.400309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:32.401613 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:32.401624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:32.402456 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:32.402464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:32.402696 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:32.403446 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:27:32.403470 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:27:32.406164 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:27:32.409722 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:32.409739 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:32.409753 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:32.410234 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:32.410691 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:32.410697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:32.410700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:32.410905 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:32.411446 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:27:32.411454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:27:32.411813 (+ 359us) client_negotiation.cc:769] Sending connection context
0809 20:27:32.412096 (+ 283us) client_negotiation.cc:240] Negotiation successful
0809 20:27:32.412753 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":298,"threads_started":1}
I20260809 20:27:32.415475 29386 heartbeater.cc:347] Connected to a master server at 127.12.78.62:42277
I20260809 20:27:32.415856 29386 heartbeater.cc:464] Registering TS with master...
I20260809 20:27:32.416680 29386 heartbeater.cc:511] Master 127.12.78.62:42277 requested a full tablet report, sending...
I20260809 20:27:32.418318 29054 ts_manager.cc:194] Registered new tserver with Master: 468cc626e28c473ebaf71388ed6b1e69 (127.12.78.2:39647)
I20260809 20:27:32.419636 29054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:45825
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:27:32.461982 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42277
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:27:32.809221 29392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:32.809512 29392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:32.809610 29392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:32.817164 29392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:32.817320 29392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:32.817546 29392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:27:32.827831 29392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:32.829311 29392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:32.830674 29392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:32.844852 29400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:32.844810 29397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:32.845207 29398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:32.849810 29392 server_base.cc:1034] running on GCE node
I20260809 20:27:32.850522 29392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:32.851629 29392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:32.852870 29392 hybrid_clock.cc:648] HybridClock initialized: now 1786307252852826 us; error 59 us; skew 500 ppm
I20260809 20:27:32.853250 29392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:32 dist-test-slave-rnrw krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307252, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:32.856842 29392 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:27:32.858676 29392 webserver.cc:460] Webserver started at http://127.12.78.3:33553/ using document root <none> and password file <none>
I20260809 20:27:32.859406 29392 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:32.859566 29392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:32.859896 29392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:27:32.863022 29392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "309996c7d4e24785a7c198edf339f563"
format_stamp: "Formatted at 2026-08-09 20:27:32 on dist-test-slave-rnrw"
server_key: "42c06bad7fdde7161dc1beaa5cc9fb9f"
server_key_iv: "fa84e20a1680c089dbc2542df14a30d7"
server_key_version: "encryptionkey@0"
I20260809 20:27:32.863826 29392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "309996c7d4e24785a7c198edf339f563"
format_stamp: "Formatted at 2026-08-09 20:27:32 on dist-test-slave-rnrw"
server_key: "42c06bad7fdde7161dc1beaa5cc9fb9f"
server_key_iv: "fa84e20a1680c089dbc2542df14a30d7"
server_key_version: "encryptionkey@0"
I20260809 20:27:32.869315 29392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 20:27:32.872874 29407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:32.874184 29392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 20:27:32.874370 29392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "309996c7d4e24785a7c198edf339f563"
format_stamp: "Formatted at 2026-08-09 20:27:32 on dist-test-slave-rnrw"
server_key: "42c06bad7fdde7161dc1beaa5cc9fb9f"
server_key_iv: "fa84e20a1680c089dbc2542df14a30d7"
server_key_version: "encryptionkey@0"
I20260809 20:27:32.874594 29392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:32.892580 29247 heartbeater.cc:503] Master 127.12.78.62:42277 was elected leader, sending a full tablet report...
I20260809 20:27:32.895160 29392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:32.899523 29392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:32.899784 29392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:32.900857 29392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:32.902256 29392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:32.902334 29392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:32.902509 29392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:32.902561 29392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:32.937861 29392 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:37971
I20260809 20:27:32.937964 29524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:37971 every 8 connection(s)
I20260809 20:27:32.939213 29392 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:27:32.941072 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 29392
I20260809 20:27:32.941277 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:27:32.941780 12600 external_mini_cluster.cc:1428] Setting key 68ea418755f7cd3c37eb948076e3d1b5
Aug 09 20:27:32 dist-test-slave-rnrw krb5kdc[29007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307252, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:32.961462 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:32.942144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:51879 (local address 127.12.78.62:42277)
0809 20:27:32.942905 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:32.942912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:32.943753 (+ 841us) server_negotiation.cc:407] Connection header received
0809 20:27:32.946118 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:32.946123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:32.946211 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:32.946338 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:32.947955 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:32.948849 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:32.950521 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:32.950927 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:32.951076 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:32.954621 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:32.954625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:32.954629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:32.954662 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:32.958090 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:32.958913 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:32.958920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:32.958923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:32.959075 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:32.959531 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:32.959536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:32.959539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:32.959828 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:32.959998 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:32.960811 (+ 813us) server_negotiation.cc:299] Negotiation successful
0809 20:27:32.961066 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":506,"threads_started":1}
I20260809 20:27:32.962455 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:32.942705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:51879)
0809 20:27:32.943597 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:32.943641 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:32.945845 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:32.946584 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:32.946592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:32.947124 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:27:32.947787 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:32.947805 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:32.949004 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:32.949008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:32.950377 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:32.950390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:32.951178 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:32.951186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:32.951414 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:32.952036 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:27:32.952084 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:27:32.954396 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:27:32.958237 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:32.958247 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:32.958266 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:32.958789 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:32.959192 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:32.959197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:32.959201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:32.959411 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:32.959971 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:27:32.959979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:27:32.960356 (+ 377us) client_negotiation.cc:769] Sending connection context
0809 20:27:32.960697 (+ 341us) client_negotiation.cc:240] Negotiation successful
0809 20:27:32.961275 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":379,"threads_started":1}
I20260809 20:27:32.963963 29525 heartbeater.cc:347] Connected to a master server at 127.12.78.62:42277
I20260809 20:27:32.964299 29525 heartbeater.cc:464] Registering TS with master...
I20260809 20:27:32.965131 29525 heartbeater.cc:511] Master 127.12.78.62:42277 requested a full tablet report, sending...
I20260809 20:27:32.966662 29054 ts_manager.cc:194] Registered new tserver with Master: 309996c7d4e24785a7c198edf339f563 (127.12.78.3:37971)
I20260809 20:27:32.967510 29054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:51879
I20260809 20:27:32.970842 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:27:32.990013 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:32.975841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40136 (local address 127.12.78.62:42277)
0809 20:27:32.976052 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:32.976058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:32.976488 (+ 430us) server_negotiation.cc:407] Connection header received
0809 20:27:32.976710 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:32.976715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:32.976798 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:32.976895 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:32.978657 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:32.979572 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:32.981096 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:32.981544 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:32.981712 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:32.984319 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:32.984324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:32.984328 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:32.984363 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:32.987170 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:32.988068 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:32.988072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:32.988074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:32.988216 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:32.988663 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:32.988666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:32.988668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:32.989005 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:32.989206 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:32.989548 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 20:27:32.989771 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 20:27:32.994833 29054 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:27:33.002213 29054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:27:33.041210 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.015250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:37971 (local address 127.0.0.1:59232)
0809 20:27:33.016900 (+ 1650us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:33.016953 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:33.017123 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:33.033291 (+ 16168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:33.033297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:33.033324 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:33.033798 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:33.033815 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.035832 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.035838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:33.038177 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:33.038202 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.040028 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.040033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:33.040254 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.040309 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 20:27:33.040459 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 20:27:33.040712 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1417,"mutex_wait_us":18,"thread_start_us":16307,"threads_started":1}
I20260809 20:27:33.041422 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.015545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.12.78.3:37971)
0809 20:27:33.032667 (+ 17122us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:33.032675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:33.032711 (+ 36us) server_negotiation.cc:407] Connection header received
0809 20:27:33.032810 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:33.032819 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:33.033058 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:33.034009 (+ 951us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:33.034297 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.035684 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.038385 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.039805 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.040115 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.040373 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:33.040802 (+ 429us) server_negotiation.cc:299] Negotiation successful
0809 20:27:33.040982 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16888,"thread_start_us":16706,"threads_started":1}
I20260809 20:27:33.042235 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.017365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:34943 (local address 127.0.0.1:60952)
0809 20:27:33.031845 (+ 14480us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:33.031864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:33.031993 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:33.032716 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:33.032721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:33.032749 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:33.033192 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:33.033206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.035079 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.035083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:33.039188 (+ 4105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:33.039201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.041203 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.041210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:33.041415 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.041482 (+ 67us) client_negotiation.cc:769] Sending connection context
0809 20:27:33.041576 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 20:27:33.041731 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14359,"mutex_wait_us":104,"thread_start_us":13402,"threads_started":1}
I20260809 20:27:33.042400 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.015420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:39647 (local address 127.0.0.1:46310)
0809 20:27:33.031068 (+ 15648us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:33.031095 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:33.031285 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:33.032386 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:33.032394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:33.032439 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:33.032928 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:33.032943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.035263 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.035269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:33.037914 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:33.037932 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.040007 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.040016 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:33.041641 (+ 1625us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.041686 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 20:27:33.041763 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 20:27:33.041894 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15501,"mutex_wait_us":130,"thread_start_us":1076,"threads_started":1}
I20260809 20:27:33.042861 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.017686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.12.78.1:34943)
0809 20:27:33.030203 (+ 12517us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:33.030218 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:33.032015 (+ 1797us) server_negotiation.cc:407] Connection header received
0809 20:27:33.032176 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:33.032188 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:33.032459 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:33.032737 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:33.033444 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.034925 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.039385 (+ 4460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.040958 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.041116 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.041927 (+ 811us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:33.042007 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 20:27:33.042218 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12231,"thread_start_us":12555,"threads_started":1}
I20260809 20:27:33.043138 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.016091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.12.78.2:39647)
0809 20:27:33.031528 (+ 15437us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:33.031540 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:33.031578 (+ 38us) server_negotiation.cc:407] Connection header received
0809 20:27:33.031704 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:33.031716 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:33.032057 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:33.033179 (+ 1122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:33.033446 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.035112 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.038119 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.039726 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.040475 (+ 749us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.040699 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:33.042639 (+ 1940us) server_negotiation.cc:299] Negotiation successful
0809 20:27:33.042816 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15208,"thread_start_us":15052,"threads_started":1}
I20260809 20:27:33.046212 29318 tablet_service.cc:1467] Processing CreateTablet for tablet 6446bfafd2374e02966e8cbf04161c8e (DEFAULT_TABLE table=test-table [id=dfa1d828f77244ec8c7de2bbd71e9662]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:27:33.046361 29180 tablet_service.cc:1467] Processing CreateTablet for tablet 6446bfafd2374e02966e8cbf04161c8e (DEFAULT_TABLE table=test-table [id=dfa1d828f77244ec8c7de2bbd71e9662]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:27:33.047523 29318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6446bfafd2374e02966e8cbf04161c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:33.046365 29457 tablet_service.cc:1467] Processing CreateTablet for tablet 6446bfafd2374e02966e8cbf04161c8e (DEFAULT_TABLE table=test-table [id=dfa1d828f77244ec8c7de2bbd71e9662]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:27:33.047804 29180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6446bfafd2374e02966e8cbf04161c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:33.048705 29457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6446bfafd2374e02966e8cbf04161c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:33.060160 29544 tablet_bootstrap.cc:492] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004: Bootstrap starting.
I20260809 20:27:33.062839 29545 tablet_bootstrap.cc:492] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69: Bootstrap starting.
I20260809 20:27:33.064014 29544 tablet_bootstrap.cc:654] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:33.064612 29546 tablet_bootstrap.cc:492] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563: Bootstrap starting.
I20260809 20:27:33.065574 29544 log.cc:824] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:33.066479 29545 tablet_bootstrap.cc:654] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:33.067763 29545 log.cc:824] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:33.067911 29546 tablet_bootstrap.cc:654] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:33.069000 29544 tablet_bootstrap.cc:492] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004: No bootstrap required, opened a new log
I20260809 20:27:33.069149 29546 log.cc:824] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:33.069293 29544 ts_tablet_manager.cc:1397] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260809 20:27:33.071164 29545 tablet_bootstrap.cc:492] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69: No bootstrap required, opened a new log
I20260809 20:27:33.071434 29545 ts_tablet_manager.cc:1397] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260809 20:27:33.072295 29546 tablet_bootstrap.cc:492] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563: No bootstrap required, opened a new log
I20260809 20:27:33.072602 29546 ts_tablet_manager.cc:1397] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260809 20:27:33.079124 29545 raft_consensus.cc:348] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } }
I20260809 20:27:33.079614 29545 raft_consensus.cc:374] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:33.079712 29545 raft_consensus.cc:729] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468cc626e28c473ebaf71388ed6b1e69, State: Initialized, Role: FOLLOWER
I20260809 20:27:33.080075 29544 raft_consensus.cc:348] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } }
I20260809 20:27:33.080467 29544 raft_consensus.cc:374] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:33.080595 29544 raft_consensus.cc:729] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 670837917ff8439386f4315d00fc8004, State: Initialized, Role: FOLLOWER
I20260809 20:27:33.080396 29545 consensus_queue.cc:260] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } }
I20260809 20:27:33.081151 29544 consensus_queue.cc:260] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } }
I20260809 20:27:33.083590 29544 ts_tablet_manager.cc:1428] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260809 20:27:33.083343 29546 raft_consensus.cc:348] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } }
I20260809 20:27:33.083760 29546 raft_consensus.cc:374] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:33.083848 29546 raft_consensus.cc:729] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309996c7d4e24785a7c198edf339f563, State: Initialized, Role: FOLLOWER
I20260809 20:27:33.084597 29386 heartbeater.cc:503] Master 127.12.78.62:42277 was elected leader, sending a full tablet report...
I20260809 20:27:33.084421 29546 consensus_queue.cc:260] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } }
I20260809 20:27:33.085364 29545 ts_tablet_manager.cc:1428] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260809 20:27:33.086210 29525 heartbeater.cc:503] Master 127.12.78.62:42277 was elected leader, sending a full tablet report...
I20260809 20:27:33.087754 29546 ts_tablet_manager.cc:1428] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260809 20:27:33.112849 29248 tablet.cc:2369] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:27:33.142655 29387 tablet.cc:2369] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:27:33.191041 29526 tablet.cc:2369] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:27:33.260274 29551 raft_consensus.cc:482] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:27:33.260632 29551 raft_consensus.cc:504] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } }
I20260809 20:27:33.262451 29551 leader_election.cc:288] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 309996c7d4e24785a7c198edf339f563 (127.12.78.3:37971), 468cc626e28c473ebaf71388ed6b1e69 (127.12.78.2:39647)
I20260809 20:27:33.269994 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.263051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:60437 (local address 127.12.78.2:39647)
0809 20:27:33.263281 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:33.263287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:33.264234 (+ 947us) server_negotiation.cc:407] Connection header received
0809 20:27:33.264471 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:33.264475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:33.264536 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:33.264665 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:33.265274 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.265989 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.268006 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.268892 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.269076 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.269200 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:33.269548 (+ 348us) server_negotiation.cc:299] Negotiation successful
0809 20:27:33.269693 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 20:27:33.269985 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.262956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:33167 (local address 127.12.78.3:37971)
0809 20:27:33.263186 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:33.263193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:33.264333 (+ 1140us) server_negotiation.cc:407] Connection header received
0809 20:27:33.264535 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:33.264565 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:33.264648 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:33.264748 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:33.265403 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.266157 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.268019 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.268902 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.269064 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.269187 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:33.269521 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 20:27:33.269668 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 20:27:33.270147 29553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.263066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:37971 (local address 127.12.78.1:33167)
0809 20:27:33.264208 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:33.264223 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:33.264386 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:33.264766 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:33.264775 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:33.264815 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:33.265224 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:33.265233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.266310 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.266315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:33.267855 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:33.267868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.269150 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.269157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:33.269320 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.269374 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 20:27:33.269480 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 20:27:33.269655 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":43,"thread_start_us":584,"threads_started":1}
I20260809 20:27:33.270197 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.263054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:39647 (local address 127.12.78.1:60437)
0809 20:27:33.264082 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:33.264100 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:33.264309 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:33.264686 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:33.264689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:33.264714 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:33.265118 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:33.265129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.266127 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.266136 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:33.267846 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:33.267858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.269201 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.269208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:33.269390 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.269414 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 20:27:33.269520 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 20:27:33.269700 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":149,"thread_start_us":579,"threads_started":1}
I20260809 20:27:33.271286 29340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6446bfafd2374e02966e8cbf04161c8e" candidate_uuid: "670837917ff8439386f4315d00fc8004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "468cc626e28c473ebaf71388ed6b1e69" is_pre_election: true
I20260809 20:27:33.271279 29477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6446bfafd2374e02966e8cbf04161c8e" candidate_uuid: "670837917ff8439386f4315d00fc8004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309996c7d4e24785a7c198edf339f563" is_pre_election: true
I20260809 20:27:33.271804 29477 raft_consensus.cc:2454] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 670837917ff8439386f4315d00fc8004 in term 0.
I20260809 20:27:33.271812 29340 raft_consensus.cc:2454] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 670837917ff8439386f4315d00fc8004 in term 0.
I20260809 20:27:33.272527 29135 leader_election.cc:302] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309996c7d4e24785a7c198edf339f563, 670837917ff8439386f4315d00fc8004; no voters:
I20260809 20:27:33.273068 29551 raft_consensus.cc:2781] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:27:33.273226 29551 raft_consensus.cc:482] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:27:33.273335 29551 raft_consensus.cc:3037] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:33.275715 29551 raft_consensus.cc:504] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } }
I20260809 20:27:33.276348 29551 leader_election.cc:288] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [CANDIDATE]: Term 1 election: Requested vote from peers 309996c7d4e24785a7c198edf339f563 (127.12.78.3:37971), 468cc626e28c473ebaf71388ed6b1e69 (127.12.78.2:39647)
I20260809 20:27:33.276943 29477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6446bfafd2374e02966e8cbf04161c8e" candidate_uuid: "670837917ff8439386f4315d00fc8004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309996c7d4e24785a7c198edf339f563"
I20260809 20:27:33.276940 29340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6446bfafd2374e02966e8cbf04161c8e" candidate_uuid: "670837917ff8439386f4315d00fc8004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "468cc626e28c473ebaf71388ed6b1e69"
I20260809 20:27:33.277209 29477 raft_consensus.cc:3037] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:33.277199 29340 raft_consensus.cc:3037] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:33.280579 29340 raft_consensus.cc:2454] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 670837917ff8439386f4315d00fc8004 in term 1.
I20260809 20:27:33.280579 29477 raft_consensus.cc:2454] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 670837917ff8439386f4315d00fc8004 in term 1.
I20260809 20:27:33.281037 29135 leader_election.cc:302] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309996c7d4e24785a7c198edf339f563, 670837917ff8439386f4315d00fc8004; no voters:
I20260809 20:27:33.281366 29551 raft_consensus.cc:2781] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:27:33.281843 29551 raft_consensus.cc:686] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [term 1 LEADER]: Becoming Leader. State: Replica: 670837917ff8439386f4315d00fc8004, State: Running, Role: LEADER
I20260809 20:27:33.282395 29551 consensus_queue.cc:237] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } }
I20260809 20:27:33.286912 29054 catalog_manager.cc:5495] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 reported cstate change: term changed from 0 to 1, leader changed from <none> to 670837917ff8439386f4315d00fc8004 (127.12.78.1). New cstate: current_term: 1 leader_uuid: "670837917ff8439386f4315d00fc8004" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } health_report { overall_health: HEALTHY } } }
I20260809 20:27:33.325680 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.319917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.12.78.1:34943)
0809 20:27:33.320129 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:33.320134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:33.320217 (+ 83us) server_negotiation.cc:407] Connection header received
0809 20:27:33.320456 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:33.320460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:33.320626 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:33.320797 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:27:33.321209 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.321916 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.323413 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.323842 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.323940 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.324431 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:27:33.325052 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:27:33.325169 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:33.325304 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 20:27:33.325378 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 20:27:33.334421 29477 raft_consensus.cc:1261] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563 [term 1 FOLLOWER]: Refusing update from remote peer 670837917ff8439386f4315d00fc8004: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:27:33.334421 29340 raft_consensus.cc:1261] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69 [term 1 FOLLOWER]: Refusing update from remote peer 670837917ff8439386f4315d00fc8004: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:27:33.335639 29555 consensus_queue.cc:1035] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [LEADER]: Connected to new peer: Peer: permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:27:33.336004 29551 consensus_queue.cc:1035] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:27:33.364086 29054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:27:33.364357 29054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:27:33.368643 29054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Sending DeleteTablet for 3 replicas of tablet 6446bfafd2374e02966e8cbf04161c8e
I20260809 20:27:33.370159 29457 tablet_service.cc:1514] Processing DeleteTablet for tablet 6446bfafd2374e02966e8cbf04161c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:27:33 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:59232
I20260809 20:27:33.370524 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 29117
I20260809 20:27:33.371513 29318 tablet_service.cc:1514] Processing DeleteTablet for tablet 6446bfafd2374e02966e8cbf04161c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:27:33 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:46310
I20260809 20:27:33.372259 29180 tablet_service.cc:1514] Processing DeleteTablet for tablet 6446bfafd2374e02966e8cbf04161c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:27:33 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:60952
I20260809 20:27:33.382964 29570 tablet_replica.cc:330] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69: stopping tablet replica
I20260809 20:27:33.384493 29570 raft_consensus.cc:2229] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:27:33.386171 29570 raft_consensus.cc:2258] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:27:33.389662 29571 tablet_replica.cc:330] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004: stopping tablet replica
I20260809 20:27:33.390120 29571 raft_consensus.cc:2229] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:27:33.390321 29570 ts_tablet_manager.cc:1905] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:27:33.390843 29571 raft_consensus.cc:2258] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:27:33.391467 29568 tablet_replica.cc:330] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563: stopping tablet replica
I20260809 20:27:33.391923 29568 raft_consensus.cc:2229] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:27:33.392515 29568 raft_consensus.cc:2258] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:27:33.395017 29571 ts_tablet_manager.cc:1905] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:27:33.395017 29568 ts_tablet_manager.cc:1905] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:27:33.399788 29570 ts_tablet_manager.cc:1918] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:27:33.400008 29570 log.cc:1196] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/6446bfafd2374e02966e8cbf04161c8e
I20260809 20:27:33.400745 29570 ts_tablet_manager.cc:1939] T 6446bfafd2374e02966e8cbf04161c8e P 468cc626e28c473ebaf71388ed6b1e69: Deleting consensus metadata
I20260809 20:27:33.402312 29568 ts_tablet_manager.cc:1918] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:27:33.402499 29568 log.cc:1196] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/6446bfafd2374e02966e8cbf04161c8e
I20260809 20:27:33.402822 29038 catalog_manager.cc:4841] TS 468cc626e28c473ebaf71388ed6b1e69 (127.12.78.2:39647): tablet 6446bfafd2374e02966e8cbf04161c8e (table test-table [id=dfa1d828f77244ec8c7de2bbd71e9662]) successfully deleted
I20260809 20:27:33.403054 29568 ts_tablet_manager.cc:1939] T 6446bfafd2374e02966e8cbf04161c8e P 309996c7d4e24785a7c198edf339f563: Deleting consensus metadata
I20260809 20:27:33.404932 29039 catalog_manager.cc:4841] TS 309996c7d4e24785a7c198edf339f563 (127.12.78.3:37971): tablet 6446bfafd2374e02966e8cbf04161c8e (table test-table [id=dfa1d828f77244ec8c7de2bbd71e9662]) successfully deleted
I20260809 20:27:33.406246 29571 ts_tablet_manager.cc:1918] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:27:33.406457 29571 log.cc:1196] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/6446bfafd2374e02966e8cbf04161c8e
I20260809 20:27:33.406924 29571 ts_tablet_manager.cc:1939] T 6446bfafd2374e02966e8cbf04161c8e P 670837917ff8439386f4315d00fc8004: Deleting consensus metadata
I20260809 20:27:33.408437 29038 catalog_manager.cc:4841] TS 670837917ff8439386f4315d00fc8004 (127.12.78.1:34943): tablet 6446bfafd2374e02966e8cbf04161c8e (table test-table [id=dfa1d828f77244ec8c7de2bbd71e9662]) successfully deleted
Aug 09 20:27:33 dist-test-slave-rnrw krb5kdc[29007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:27:33.417651 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.371859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.12.78.1:34943)
0809 20:27:33.372090 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:33.372097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:33.387572 (+ 15475us) server_negotiation.cc:407] Connection header received
0809 20:27:33.388876 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:33.388882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:33.388984 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:33.389069 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:33.395870 (+ 6801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.396978 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.403279 (+ 6301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.404021 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.404173 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.409806 (+ 5633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:33.409821 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:33.409840 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:33.409887 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:33.414061 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:33.415049 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:33.415059 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:33.415063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:33.415258 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:33.415766 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:33.415772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:33.415776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:33.416281 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:33.416658 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:33.416997 (+ 339us) server_negotiation.cc:299] Negotiation successful
0809 20:27:33.417275 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 20:27:33.418493 29242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:33.520998 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 29117
I20260809 20:27:33.542399 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 29253
Aug 09 20:27:33 dist-test-slave-rnrw krb5kdc[29007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:27:33.559876 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.543301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.12.78.2:39647)
0809 20:27:33.543472 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:33.543476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:33.543608 (+ 132us) server_negotiation.cc:407] Connection header received
0809 20:27:33.543780 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:33.543784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:33.543867 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:33.543956 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:33.545346 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.546078 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.547546 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.547963 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.548116 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.552924 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:33.552940 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:33.552979 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:33.553025 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:33.556513 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:33.557412 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:33.557421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:33.557424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:33.557570 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:33.558019 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:33.558025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:33.558029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:33.558466 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:33.558813 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:33.559163 (+ 350us) server_negotiation.cc:299] Negotiation successful
0809 20:27:33.559415 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 20:27:33.560745 29381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:33.655527 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 29253
I20260809 20:27:33.674856 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 29392
Aug 09 20:27:33 dist-test-slave-rnrw krb5kdc[29007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:27:33.691347 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:33.675708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.12.78.3:37971)
0809 20:27:33.675896 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:33.675903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:33.675978 (+ 75us) server_negotiation.cc:407] Connection header received
0809 20:27:33.676212 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:33.676216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:33.676291 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:33.676412 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:33.677999 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.678742 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.680151 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:33.680581 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:33.680735 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:33.685382 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:33.685400 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:33.685416 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:33.685454 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:33.688127 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:33.689150 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:33.689167 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:33.689170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:33.689327 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:33.689740 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:33.689744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:33.689746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:33.690095 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:33.690407 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:33.690745 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 20:27:33.690977 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 20:27:33.692073 29520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:33.782545 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 29392
I20260809 20:27:33.801043 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 29020
I20260809 20:27:33.802238 29085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:33.914332 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 29020
I20260809 20:27:53.939739 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:42277
--webserver_interface=127.12.78.62
--webserver_port=34291
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:42277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:27:54.300448 29576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:54.300859 29576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:54.300958 29576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:54.308977 29576 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 20:27:54.309135 29576 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 20:27:54.309194 29576 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 20:27:54.309249 29576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:27:54.309304 29576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:54.309389 29576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:54.309443 29576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:27:54.309499 29576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:27:54.319993 29576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:42277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:42277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=34291
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:54.321382 29576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:54.322742 29576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:54.349812 29585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:54.349908 29582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:54.349936 29583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:54.350210 29576 server_base.cc:1034] running on GCE node
I20260809 20:27:54.352011 29576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:54.353981 29576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:54.355229 29576 hybrid_clock.cc:648] HybridClock initialized: now 1786307274355124 us; error 70 us; skew 500 ppm
I20260809 20:27:54.355592 29576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:54 dist-test-slave-rnrw krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307274, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:54.365316 29576 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:27:54.367264 29576 webserver.cc:460] Webserver started at http://127.12.78.62:34291/ using document root <none> and password file <none>
I20260809 20:27:54.368156 29576 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:54.368311 29576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:54.374908 29576 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260809 20:27:54.378250 29592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:54.379804 29576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:27:54.380023 29576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "a9041503701d4e079f66889a0a249e58"
format_stamp: "Formatted at 2026-08-09 20:27:31 on dist-test-slave-rnrw"
server_key: "99391ddb2d1e8d28ff12284d6f8f0fd2"
server_key_iv: "6a1ac375fc5c777e617237ffcfd84307"
server_key_version: "encryptionkey@0"
I20260809 20:27:54.381281 29576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:54.406009 29576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:54.410979 29576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:54.411312 29576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:54.432361 29576 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:42277
I20260809 20:27:54.432413 29644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:42277 every 8 connection(s)
I20260809 20:27:54.434280 29576 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:27:54.442230 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 29576
I20260809 20:27:54.442435 29645 sys_catalog.cc:263] Verifying existing consensus state
I20260809 20:27:54.443390 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:34943
--local_ip_for_outbound_sockets=127.12.78.1
--tserver_master_addrs=127.12.78.62:42277
--webserver_port=41487
--webserver_interface=127.12.78.1
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:27:54.448181 29645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Bootstrap starting.
I20260809 20:27:54.471467 29645 log.cc:824] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:54.485967 29645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 20:27:54.486712 29645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Bootstrap complete.
I20260809 20:27:54.497674 29645 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } }
I20260809 20:27:54.498669 29645 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9041503701d4e079f66889a0a249e58, State: Initialized, Role: FOLLOWER
I20260809 20:27:54.499339 29645 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } }
I20260809 20:27:54.499624 29645 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:27:54.499770 29645 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:27:54.499984 29645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 1 FOLLOWER]: Advancing to term 2
I20260809 20:27:54.504283 29645 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } }
I20260809 20:27:54.504971 29645 leader_election.cc:302] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9041503701d4e079f66889a0a249e58; no voters:
I20260809 20:27:54.505729 29645 leader_election.cc:288] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 20:27:54.505883 29649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 20:27:54.507311 29649 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 2 LEADER]: Becoming Leader. State: Replica: a9041503701d4e079f66889a0a249e58, State: Running, Role: LEADER
I20260809 20:27:54.508006 29649 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } }
I20260809 20:27:54.508347 29645 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:27:54.511186 29652 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9041503701d4e079f66889a0a249e58. Latest consensus state: current_term: 2 leader_uuid: "a9041503701d4e079f66889a0a249e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } } }
I20260809 20:27:54.511242 29651 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a9041503701d4e079f66889a0a249e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } } }
I20260809 20:27:54.511811 29651 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: This master's current role is: LEADER
I20260809 20:27:54.511837 29652 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: This master's current role is: LEADER
I20260809 20:27:54.526762 29658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:27:54.533841 29658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:27:54.535022 29658 catalog_manager.cc:1259] Loaded cluster ID: f0b8b1bd09644b7aa8dc00c67995a08d
I20260809 20:27:54.535154 29658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:27:54.538260 29658 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:27:54.539613 29658 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Loaded TSK: 0
I20260809 20:27:54.566566 29658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Generated new TSK 1
I20260809 20:27:54.567087 29658 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 20:27:54.854849 29647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:54.855224 29647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:54.855406 29647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:54.863225 29647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:54.863444 29647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:54.863690 29647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:27:54.874447 29647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:34943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=41487
--tserver_master_addrs=127.12.78.62:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:54.875949 29647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:54.877482 29647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:54.891734 29677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:54.891708 29675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:54.892892 29674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:54.893302 29647 server_base.cc:1034] running on GCE node
I20260809 20:27:54.893946 29647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:54.895062 29647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:54.896358 29647 hybrid_clock.cc:648] HybridClock initialized: now 1786307274896291 us; error 57 us; skew 500 ppm
I20260809 20:27:54.896838 29647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:54 dist-test-slave-rnrw krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307274, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:54.900234 29647 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:27:54.902092 29647 webserver.cc:460] Webserver started at http://127.12.78.1:41487/ using document root <none> and password file <none>
I20260809 20:27:54.902829 29647 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:54.902987 29647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:54.908999 29647 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 20:27:54.912273 29684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:54.913784 29647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:27:54.913985 29647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "670837917ff8439386f4315d00fc8004"
format_stamp: "Formatted at 2026-08-09 20:27:31 on dist-test-slave-rnrw"
server_key: "40c9e00144d1e680b42d1c26fbb1db1f"
server_key_iv: "5dfc7bf660a38693fbd3ba48015167e6"
server_key_version: "encryptionkey@0"
I20260809 20:27:54.915189 29647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:54.932821 29647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:54.937340 29647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:54.937666 29647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:54.938809 29647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:54.940255 29647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:54.940385 29647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:54.940512 29647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:54.940624 29647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 20:27:54.976145 29647 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:34943
I20260809 20:27:54.976243 29797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:34943 every 8 connection(s)
I20260809 20:27:54.977720 29647 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:27:54.978525 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 29647
I20260809 20:27:54.979431 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:39647
--local_ip_for_outbound_sockets=127.12.78.2
--tserver_master_addrs=127.12.78.62:42277
--webserver_port=35361
--webserver_interface=127.12.78.2
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 20:27:54 dist-test-slave-rnrw krb5kdc[29007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307274, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:55.006033 29800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:54.981017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:59477 (local address 127.12.78.62:42277)
0809 20:27:54.981715 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:54.981735 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:54.982422 (+ 687us) server_negotiation.cc:407] Connection header received
0809 20:27:54.984738 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:54.984761 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:54.985275 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:54.985709 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:54.987562 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:54.989736 (+ 2174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:54.992322 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:54.993099 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:54.993325 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:54.997659 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:54.997669 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:54.997688 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:54.997741 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:55.000985 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:55.001898 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:55.001906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:55.001913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:55.002102 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:55.002586 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:55.002593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:55.002599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:55.003138 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:55.003377 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:55.004664 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0809 20:27:55.005015 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":271,"threads_started":1}
I20260809 20:27:55.006832 29801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:54.981278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:59477)
0809 20:27:54.982209 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:54.982264 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:54.984061 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:54.985968 (+ 1907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:54.985977 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:54.986669 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:27:54.987283 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:54.987303 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:54.990082 (+ 2779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:54.990091 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:54.992164 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:54.992180 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:54.993386 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:54.993395 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:54.993688 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:54.994449 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:27:54.994475 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:27:54.997306 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:27:55.001200 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:55.001211 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:55.001234 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:55.001729 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:55.002241 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:55.002246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:55.002249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:55.002465 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:55.003384 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:27:55.003397 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:27:55.003980 (+ 583us) client_negotiation.cc:769] Sending connection context
0809 20:27:55.004400 (+ 420us) client_negotiation.cc:240] Negotiation successful
0809 20:27:55.005178 (+ 778us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":405,"threads_started":1}
I20260809 20:27:55.010121 29798 heartbeater.cc:347] Connected to a master server at 127.12.78.62:42277
I20260809 20:27:55.010483 29798 heartbeater.cc:464] Registering TS with master...
I20260809 20:27:55.011272 29798 heartbeater.cc:511] Master 127.12.78.62:42277 requested a full tablet report, sending...
I20260809 20:27:55.014217 29609 ts_manager.cc:194] Registered new tserver with Master: 670837917ff8439386f4315d00fc8004 (127.12.78.1:34943)
I20260809 20:27:55.016602 29609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:59477
W20260809 20:27:55.347177 29802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:55.347519 29802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:55.347689 29802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:55.355377 29802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:55.355506 29802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:55.355710 29802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:27:55.366016 29802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:39647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=35361
--tserver_master_addrs=127.12.78.62:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:55.367491 29802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:55.369138 29802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:55.384600 29810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:55.384634 29808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:55.384469 29807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:55.390666 29802 server_base.cc:1034] running on GCE node
I20260809 20:27:55.391433 29802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:55.392685 29802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:55.393937 29802 hybrid_clock.cc:648] HybridClock initialized: now 1786307275393870 us; error 67 us; skew 500 ppm
I20260809 20:27:55.394304 29802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:55 dist-test-slave-rnrw krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307275, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:55.397864 29802 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:27:55.399581 29802 webserver.cc:460] Webserver started at http://127.12.78.2:35361/ using document root <none> and password file <none>
I20260809 20:27:55.400341 29802 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:55.400509 29802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:55.406386 29802 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260809 20:27:55.409412 29817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:55.411124 29802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:27:55.411332 29802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "468cc626e28c473ebaf71388ed6b1e69"
format_stamp: "Formatted at 2026-08-09 20:27:32 on dist-test-slave-rnrw"
server_key: "1c607ceec687b50da5002c8943be9c7e"
server_key_iv: "3b827a162327df7e574ca83e0a675c19"
server_key_version: "encryptionkey@0"
I20260809 20:27:55.412719 29802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:55.432394 29802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:55.437194 29802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:55.437497 29802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:55.438602 29802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:55.440141 29802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:55.440230 29802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:55.440390 29802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:55.440479 29802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:55.476352 29802 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:39647
I20260809 20:27:55.476454 29930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:39647 every 8 connection(s)
I20260809 20:27:55.477864 29802 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:27:55.484112 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 29802
I20260809 20:27:55.485033 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:37971
--local_ip_for_outbound_sockets=127.12.78.3
--tserver_master_addrs=127.12.78.62:42277
--webserver_port=33553
--webserver_interface=127.12.78.3
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 20:27:55 dist-test-slave-rnrw krb5kdc[29007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307275, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:55.503425 29800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:55.480759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:52295 (local address 127.12.78.62:42277)
0809 20:27:55.480993 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:55.480998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:55.482137 (+ 1139us) server_negotiation.cc:407] Connection header received
0809 20:27:55.484334 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:55.484340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:55.484609 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:55.484719 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:55.487043 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:55.487786 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:55.490440 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:55.490867 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:55.490979 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:55.495914 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:55.495920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:55.495923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:55.495962 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:55.499688 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:55.500611 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:55.500620 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:55.500626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:55.500823 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:55.501340 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:55.501348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:55.501354 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:55.501765 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:55.501914 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:55.502675 (+ 761us) server_negotiation.cc:299] Negotiation successful
0809 20:27:55.502969 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 20:27:55.504475 29933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:55.481203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:52295)
0809 20:27:55.481935 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:55.481973 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:55.483961 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:55.485061 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:55.485076 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:55.485914 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:27:55.486828 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:55.486859 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:55.487985 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:55.487993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:55.490268 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:55.490288 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:55.491143 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:55.491153 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:55.491517 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:55.492332 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:27:55.492375 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:27:55.495596 (+ 3221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:27:55.499876 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:55.499884 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:55.499899 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:55.500420 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:55.500973 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:55.500976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:55.500979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:55.501208 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:55.501905 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:27:55.501911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:27:55.502280 (+ 369us) client_negotiation.cc:769] Sending connection context
0809 20:27:55.502554 (+ 274us) client_negotiation.cc:240] Negotiation successful
0809 20:27:55.503247 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":319,"threads_started":1}
I20260809 20:27:55.506202 29931 heartbeater.cc:347] Connected to a master server at 127.12.78.62:42277
I20260809 20:27:55.506526 29931 heartbeater.cc:464] Registering TS with master...
I20260809 20:27:55.507316 29931 heartbeater.cc:511] Master 127.12.78.62:42277 requested a full tablet report, sending...
I20260809 20:27:55.508958 29609 ts_manager.cc:194] Registered new tserver with Master: 468cc626e28c473ebaf71388ed6b1e69 (127.12.78.2:39647)
I20260809 20:27:55.509807 29609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:52295
W20260809 20:27:55.852244 29934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:27:55.852535 29934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:27:55.852710 29934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:27:55.860309 29934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:27:55.860438 29934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:27:55.860641 29934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:27:55.871012 29934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:37971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=33553
--tserver_master_addrs=127.12.78.62:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:27:55.872496 29934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:27:55.874037 29934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:27:55.887806 29940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:55.887874 29941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:27:55.887853 29943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:27:55.889194 29934 server_base.cc:1034] running on GCE node
I20260809 20:27:55.889899 29934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:27:55.890846 29934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:27:55.892122 29934 hybrid_clock.cc:648] HybridClock initialized: now 1786307275892050 us; error 43 us; skew 500 ppm
I20260809 20:27:55.892464 29934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:27:55 dist-test-slave-rnrw krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307275, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:27:55.895793 29934 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:27:55.897576 29934 webserver.cc:460] Webserver started at http://127.12.78.3:33553/ using document root <none> and password file <none>
I20260809 20:27:55.898276 29934 fs_manager.cc:359] Metadata directory not provided
I20260809 20:27:55.898440 29934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:27:55.904145 29934 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 20:27:55.907114 29950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:55.908790 29934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 20:27:55.908943 29934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "309996c7d4e24785a7c198edf339f563"
format_stamp: "Formatted at 2026-08-09 20:27:32 on dist-test-slave-rnrw"
server_key: "42c06bad7fdde7161dc1beaa5cc9fb9f"
server_key_iv: "fa84e20a1680c089dbc2542df14a30d7"
server_key_version: "encryptionkey@0"
I20260809 20:27:55.909971 29934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:27:55.923607 29934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:27:55.928107 29934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:27:55.928347 29934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:27:55.929486 29934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:27:55.930820 29934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:27:55.930920 29934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:55.930992 29934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:27:55.931061 29934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:27:55.966285 29934 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:37971
I20260809 20:27:55.966374 30065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:37971 every 8 connection(s)
I20260809 20:27:55.967837 29934 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:27:55.970283 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 29934
Aug 09 20:27:55 dist-test-slave-rnrw krb5kdc[29007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307275, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:27:55.991196 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:55.971425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.12.78.62:42277)
0809 20:27:55.972895 (+ 1470us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:55.972905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:55.974958 (+ 2053us) server_negotiation.cc:407] Connection header received
0809 20:27:55.975143 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:55.975146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:55.975193 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:55.975304 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:55.977357 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:55.978345 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:55.979810 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:55.980361 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:55.980496 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:55.983983 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:55.983989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:55.983995 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:55.984040 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:55.987975 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:55.988807 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:55.988813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:55.988816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:55.988978 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:55.989414 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:55.989419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:55.989422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:55.989860 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:55.990057 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:55.990469 (+ 412us) server_negotiation.cc:299] Negotiation successful
0809 20:27:55.990740 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1318,"thread_start_us":1169,"threads_started":1}
I20260809 20:27:55.991688 29800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:55.970465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:36027 (local address 127.12.78.62:42277)
0809 20:27:55.970698 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:55.970705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:55.972222 (+ 1517us) server_negotiation.cc:407] Connection header received
0809 20:27:55.974437 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:55.974447 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:55.974555 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:55.974682 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:55.976925 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:55.978077 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:55.980400 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:55.980942 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:55.981076 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:55.984871 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:55.984875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:55.984878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:55.984918 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:55.988126 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:55.989113 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:55.989119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:55.989122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:55.989259 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:55.989797 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:55.989800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:55.989802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:55.990163 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:55.990319 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:55.991246 (+ 927us) server_negotiation.cc:299] Negotiation successful
0809 20:27:55.991409 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 20:27:55.993163 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:55.970950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:36027)
0809 20:27:55.972001 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:55.972076 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:55.974147 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:55.974974 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:55.974986 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:55.975818 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:27:55.976719 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:55.976751 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:55.978292 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:55.978297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:55.980254 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:55.980269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:55.981185 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:55.981192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:55.981564 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:55.982194 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:27:55.982217 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:27:55.984654 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:27:55.988277 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:55.988285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:55.988298 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:55.988948 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:55.989404 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:27:55.989409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:27:55.989412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:27:55.989673 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:27:55.990325 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:27:55.990336 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:27:55.990817 (+ 481us) client_negotiation.cc:769] Sending connection context
0809 20:27:55.991156 (+ 339us) client_negotiation.cc:240] Negotiation successful
0809 20:27:55.991798 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":376,"threads_started":1}
I20260809 20:27:55.994642 30066 heartbeater.cc:347] Connected to a master server at 127.12.78.62:42277
I20260809 20:27:55.994982 30066 heartbeater.cc:464] Registering TS with master...
I20260809 20:27:55.995714 30066 heartbeater.cc:511] Master 127.12.78.62:42277 requested a full tablet report, sending...
I20260809 20:27:55.997371 29608 ts_manager.cc:194] Registered new tserver with Master: 309996c7d4e24785a7c198edf339f563 (127.12.78.3:37971)
I20260809 20:27:55.998265 29608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:36027
I20260809 20:27:56.007206 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:27:56.014796 29800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.008270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.12.78.62:42277)
0809 20:27:56.008509 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:56.008513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:56.009029 (+ 516us) server_negotiation.cc:407] Connection header received
0809 20:27:56.009313 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:56.009317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:56.009465 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:56.009652 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:27:56.010109 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.010880 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.012437 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.012950 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.013093 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.013735 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:27:56.014066 (+ 331us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 20:27:56.014488 (+ 422us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52582: token expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260809 20:27:56.014971 29800 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52582: token expired
W20260809 20:27:56.015544 29532 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.78.62:42277)
I20260809 20:27:56.020603 29798 heartbeater.cc:503] Master 127.12.78.62:42277 was elected leader, sending a full tablet report...
I20260809 20:27:56.029306 29800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.016121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.12.78.62:42277)
0809 20:27:56.016324 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:56.016330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:56.016431 (+ 101us) server_negotiation.cc:407] Connection header received
0809 20:27:56.016647 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:56.016652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:56.016740 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:56.016843 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:56.018261 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.018997 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.020511 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.021051 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.021211 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.023942 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:56.023948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:56.023953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:56.023994 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:56.026519 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:56.027254 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:56.027259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:56.027263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:56.027390 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:56.027788 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:56.027794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:56.027798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:56.028127 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:56.028388 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:56.028763 (+ 375us) server_negotiation.cc:299] Negotiation successful
0809 20:27:56.028987 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 20:27:56.033286 29608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:27:56.040757 29608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:27:56.066268 30076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.052617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:37971 (local address 127.0.0.1:37412)
0809 20:27:56.053559 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:56.053611 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:56.053823 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:56.055503 (+ 1680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:56.055519 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:56.055567 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:56.056270 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.056286 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.058910 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.058920 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.062215 (+ 3295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.062246 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.064886 (+ 2640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.064896 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.065200 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.065288 (+ 88us) client_negotiation.cc:769] Sending connection context
0809 20:27:56.065441 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 20:27:56.065704 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":1054,"threads_started":1}
I20260809 20:27:56.068120 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.052829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.12.78.3:37971)
0809 20:27:56.054582 (+ 1753us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:56.054588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:56.054615 (+ 27us) server_negotiation.cc:407] Connection header received
0809 20:27:56.054690 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:56.054695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:56.055201 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:56.055485 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:56.056518 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.058695 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.062517 (+ 3822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.064561 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.066921 (+ 2360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.067279 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:56.067371 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 20:27:56.067629 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1598,"thread_start_us":1431,"threads_started":1}
I20260809 20:27:56.069548 30078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.052794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:34943 (local address 127.0.0.1:59520)
0809 20:27:56.056080 (+ 3286us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:56.056094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:56.056255 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:56.057322 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:56.057328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:56.057356 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:56.058529 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.058547 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.061755 (+ 3208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.061760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.064717 (+ 2957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.064740 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.068775 (+ 4035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.068787 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.068978 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.069009 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 20:27:56.069091 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 20:27:56.069222 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3167,"thread_start_us":2993,"threads_started":1}
I20260809 20:27:56.070174 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.053480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59520 (local address 127.12.78.1:34943)
0809 20:27:56.055445 (+ 1965us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:56.055463 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:56.056397 (+ 934us) server_negotiation.cc:407] Connection header received
0809 20:27:56.056534 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:56.056565 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:56.056829 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:56.057049 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:56.058807 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.060879 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.065010 (+ 4131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.068149 (+ 3139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.069033 (+ 884us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.069282 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:56.069358 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 20:27:56.069549 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1782,"thread_start_us":1610,"threads_started":1}
I20260809 20:27:56.072964 30000 tablet_service.cc:1467] Processing CreateTablet for tablet aec4ef95717049c2b8b681f3bcec21d1 (DEFAULT_TABLE table=test-table [id=3251ca1bed6e4e30b9b3622da2d7588b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:27:56.073947 29732 tablet_service.cc:1467] Processing CreateTablet for tablet aec4ef95717049c2b8b681f3bcec21d1 (DEFAULT_TABLE table=test-table [id=3251ca1bed6e4e30b9b3622da2d7588b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:27:56.074565 30000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec4ef95717049c2b8b681f3bcec21d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:56.075407 29732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec4ef95717049c2b8b681f3bcec21d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:56.076714 30076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.056266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:39647 (local address 127.0.0.1:34998)
0809 20:27:56.066563 (+ 10297us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:56.066582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:56.066701 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:56.071562 (+ 4861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:56.071567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:56.071594 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:56.071776 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.071790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.073235 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.073244 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.074603 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.074624 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.075891 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.075898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.076115 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.076149 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 20:27:56.076265 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 20:27:56.076405 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10184,"thread_start_us":12541,"threads_started":1}
I20260809 20:27:56.076858 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.056525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.12.78.2:39647)
0809 20:27:56.070949 (+ 14424us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:56.070957 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:56.070986 (+ 29us) server_negotiation.cc:407] Connection header received
0809 20:27:56.071086 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:56.071098 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:56.071361 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:56.071573 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:56.071930 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.073109 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.074762 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.075732 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.075878 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.076065 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:56.076316 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 20:27:56.076478 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14256,"thread_start_us":16678,"threads_started":1}
I20260809 20:27:56.080746 29866 tablet_service.cc:1467] Processing CreateTablet for tablet aec4ef95717049c2b8b681f3bcec21d1 (DEFAULT_TABLE table=test-table [id=3251ca1bed6e4e30b9b3622da2d7588b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:27:56.082141 29866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec4ef95717049c2b8b681f3bcec21d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:27:56.088534 30082 tablet_bootstrap.cc:492] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563: Bootstrap starting.
I20260809 20:27:56.092228 30083 tablet_bootstrap.cc:492] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004: Bootstrap starting.
I20260809 20:27:56.093124 30082 tablet_bootstrap.cc:654] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:56.094434 30082 log.cc:824] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:56.095692 30083 tablet_bootstrap.cc:654] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:56.096804 30083 log.cc:824] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:56.097805 30085 tablet_bootstrap.cc:492] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69: Bootstrap starting.
I20260809 20:27:56.097949 30082 tablet_bootstrap.cc:492] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563: No bootstrap required, opened a new log
I20260809 20:27:56.098263 30082 ts_tablet_manager.cc:1397] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 20:27:56.102447 30085 tablet_bootstrap.cc:654] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69: Neither blocks nor log segments found. Creating new log.
I20260809 20:27:56.103619 30083 tablet_bootstrap.cc:492] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004: No bootstrap required, opened a new log
I20260809 20:27:56.103932 30083 ts_tablet_manager.cc:1397] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260809 20:27:56.105058 30085 log.cc:824] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69: Log is configured to *not* fsync() on all Append() calls
I20260809 20:27:56.106859 30082 raft_consensus.cc:348] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } }
I20260809 20:27:56.107234 30082 raft_consensus.cc:374] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:56.107362 30082 raft_consensus.cc:729] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309996c7d4e24785a7c198edf339f563, State: Initialized, Role: FOLLOWER
I20260809 20:27:56.107990 30082 consensus_queue.cc:260] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } }
I20260809 20:27:56.108371 30085 tablet_bootstrap.cc:492] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69: No bootstrap required, opened a new log
I20260809 20:27:56.108714 30085 ts_tablet_manager.cc:1397] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260809 20:27:56.109287 30066 heartbeater.cc:503] Master 127.12.78.62:42277 was elected leader, sending a full tablet report...
I20260809 20:27:56.109896 30082 ts_tablet_manager.cc:1428] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260809 20:27:56.114203 30083 raft_consensus.cc:348] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } }
I20260809 20:27:56.114699 30083 raft_consensus.cc:374] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:56.114807 30083 raft_consensus.cc:729] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 670837917ff8439386f4315d00fc8004, State: Initialized, Role: FOLLOWER
I20260809 20:27:56.115564 30083 consensus_queue.cc:260] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } }
I20260809 20:27:56.117600 30083 ts_tablet_manager.cc:1428] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260809 20:27:56.119639 30085 raft_consensus.cc:348] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } }
I20260809 20:27:56.120137 30085 raft_consensus.cc:374] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:27:56.120265 30085 raft_consensus.cc:729] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468cc626e28c473ebaf71388ed6b1e69, State: Initialized, Role: FOLLOWER
I20260809 20:27:56.120959 30085 consensus_queue.cc:260] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } }
I20260809 20:27:56.122017 29931 heartbeater.cc:503] Master 127.12.78.62:42277 was elected leader, sending a full tablet report...
I20260809 20:27:56.122478 30085 ts_tablet_manager.cc:1428] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
W20260809 20:27:56.219491 30067 tablet.cc:2369] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:27:56.230304 29932 tablet.cc:2369] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:27:56.230564 29799 tablet.cc:2369] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:27:56.479624 30090 raft_consensus.cc:482] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:27:56.480014 30090 raft_consensus.cc:504] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } }
I20260809 20:27:56.482156 30090 leader_election.cc:288] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 309996c7d4e24785a7c198edf339f563 (127.12.78.3:37971), 670837917ff8439386f4315d00fc8004 (127.12.78.1:34943)
I20260809 20:27:56.490108 30088 raft_consensus.cc:482] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:27:56.490513 30088 raft_consensus.cc:504] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } }
I20260809 20:27:56.491802 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.482381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:33097 (local address 127.12.78.3:37971)
0809 20:27:56.482587 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:56.482593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:56.483722 (+ 1129us) server_negotiation.cc:407] Connection header received
0809 20:27:56.484036 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:56.484042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:56.484126 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:56.484201 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:56.485155 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.486228 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.489389 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.490683 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.490834 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.490967 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:56.491345 (+ 378us) server_negotiation.cc:299] Negotiation successful
0809 20:27:56.491477 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 20:27:56.491988 30092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.482409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:37971 (local address 127.12.78.2:33097)
0809 20:27:56.483588 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:56.483613 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:56.483846 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:56.484441 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:56.484446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:56.484471 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:56.484971 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.484984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.486371 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.486380 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.489199 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.489215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.490983 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.490992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.491168 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.491194 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 20:27:56.491305 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 20:27:56.491448 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":124,"thread_start_us":1357,"threads_started":1}
I20260809 20:27:56.492723 30088 leader_election.cc:288] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 670837917ff8439386f4315d00fc8004 (127.12.78.1:34943), 468cc626e28c473ebaf71388ed6b1e69 (127.12.78.2:39647)
I20260809 20:27:56.493366 30021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec4ef95717049c2b8b681f3bcec21d1" candidate_uuid: "468cc626e28c473ebaf71388ed6b1e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309996c7d4e24785a7c198edf339f563" is_pre_election: true
I20260809 20:27:56.493952 30021 raft_consensus.cc:2454] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 468cc626e28c473ebaf71388ed6b1e69 in term 0.
I20260809 20:27:56.494877 29821 leader_election.cc:302] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309996c7d4e24785a7c198edf339f563, 468cc626e28c473ebaf71388ed6b1e69; no voters:
I20260809 20:27:56.495512 30090 raft_consensus.cc:2781] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:27:56.495677 30090 raft_consensus.cc:482] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:27:56.495833 30090 raft_consensus.cc:3037] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:56.496224 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.482820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:50961 (local address 127.12.78.1:34943)
0809 20:27:56.483027 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:56.483033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:56.484239 (+ 1206us) server_negotiation.cc:407] Connection header received
0809 20:27:56.484578 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:56.484588 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:56.484670 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:56.484781 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:56.485654 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.486814 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.489647 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.494821 (+ 5174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.495002 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.495126 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:56.495676 (+ 550us) server_negotiation.cc:299] Negotiation successful
0809 20:27:56.495883 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 20:27:56.497771 30091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.482672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:34943 (local address 127.12.78.2:50961)
0809 20:27:56.484111 (+ 1439us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:56.484128 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:56.484299 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:56.484797 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:56.484802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:56.484833 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:56.485400 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.485412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.486981 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.486987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.489482 (+ 2495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.489493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.495090 (+ 5597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.495096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.495350 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.495390 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 20:27:56.497084 (+ 1694us) client_negotiation.cc:240] Negotiation successful
0809 20:27:56.497261 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1319,"mutex_wait_us":3,"thread_start_us":578,"threads_started":1}
I20260809 20:27:56.499157 29752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec4ef95717049c2b8b681f3bcec21d1" candidate_uuid: "468cc626e28c473ebaf71388ed6b1e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "670837917ff8439386f4315d00fc8004" is_pre_election: true
I20260809 20:27:56.499785 29752 raft_consensus.cc:2454] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 468cc626e28c473ebaf71388ed6b1e69 in term 0.
I20260809 20:27:56.500409 30090 raft_consensus.cc:504] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } }
I20260809 20:27:56.501423 30090 leader_election.cc:288] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [CANDIDATE]: Term 1 election: Requested vote from peers 309996c7d4e24785a7c198edf339f563 (127.12.78.3:37971), 670837917ff8439386f4315d00fc8004 (127.12.78.1:34943)
I20260809 20:27:56.501873 30021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec4ef95717049c2b8b681f3bcec21d1" candidate_uuid: "468cc626e28c473ebaf71388ed6b1e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309996c7d4e24785a7c198edf339f563"
I20260809 20:27:56.502105 30021 raft_consensus.cc:3037] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:56.503243 29752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec4ef95717049c2b8b681f3bcec21d1" candidate_uuid: "468cc626e28c473ebaf71388ed6b1e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "670837917ff8439386f4315d00fc8004"
I20260809 20:27:56.503479 29752 raft_consensus.cc:3037] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:27:56.503590 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.493075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:34943 (local address 127.12.78.3:39675)
0809 20:27:56.493970 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:56.494006 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:56.494144 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:56.496943 (+ 2799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:56.496949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:56.496999 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:56.497205 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.497216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.498811 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.498817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.501004 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.501020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.502753 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.502759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.502990 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.503033 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 20:27:56.503103 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 20:27:56.503273 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736}
I20260809 20:27:56.504529 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.493164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:39675 (local address 127.12.78.1:34943)
0809 20:27:56.496641 (+ 3477us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:56.496648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:56.496667 (+ 19us) server_negotiation.cc:407] Connection header received
0809 20:27:56.496727 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:56.496734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:56.496800 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:56.496921 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:56.497400 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.498623 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.501244 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.502554 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.503839 (+ 1285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.503961 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:56.504019 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 20:27:56.504187 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3323,"thread_start_us":8918,"threads_started":1}
I20260809 20:27:56.505424 29751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec4ef95717049c2b8b681f3bcec21d1" candidate_uuid: "309996c7d4e24785a7c198edf339f563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "670837917ff8439386f4315d00fc8004" is_pre_election: true
I20260809 20:27:56.507197 30021 raft_consensus.cc:2454] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 468cc626e28c473ebaf71388ed6b1e69 in term 1.
I20260809 20:27:56.507220 29752 raft_consensus.cc:2454] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 468cc626e28c473ebaf71388ed6b1e69 in term 1.
I20260809 20:27:56.507706 29820 leader_election.cc:302] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 468cc626e28c473ebaf71388ed6b1e69, 670837917ff8439386f4315d00fc8004; no voters:
I20260809 20:27:56.508119 30090 raft_consensus.cc:2781] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:27:56.508875 30090 raft_consensus.cc:686] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [term 1 LEADER]: Becoming Leader. State: Replica: 468cc626e28c473ebaf71388ed6b1e69, State: Running, Role: LEADER
I20260809 20:27:56.509537 30090 consensus_queue.cc:237] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } }
I20260809 20:27:56.513775 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.494462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:35569 (local address 127.12.78.2:39647)
0809 20:27:56.494655 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:56.494659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:56.502938 (+ 8279us) server_negotiation.cc:407] Connection header received
0809 20:27:56.503243 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:56.503248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:56.503326 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:56.503390 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:27:56.508007 (+ 4617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.509128 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.511408 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.512391 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.512569 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.512675 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:56.513303 (+ 628us) server_negotiation.cc:299] Negotiation successful
0809 20:27:56.513449 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 20:27:56.513814 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.493619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:39647 (local address 127.12.78.3:35569)
0809 20:27:56.502836 (+ 9217us) negotiation.cc:107] Waiting for socket to connect
0809 20:27:56.502858 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:27:56.503058 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:27:56.503894 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:27:56.503899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:27:56.503931 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:27:56.507775 (+ 3844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.507789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.509280 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.509285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.511265 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:27:56.511281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.512735 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.512743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:27:56.512916 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.512951 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 20:27:56.513103 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 20:27:56.513454 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9057,"thread_start_us":8799,"threads_started":1}
I20260809 20:27:56.514777 29886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec4ef95717049c2b8b681f3bcec21d1" candidate_uuid: "309996c7d4e24785a7c198edf339f563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "468cc626e28c473ebaf71388ed6b1e69" is_pre_election: true
I20260809 20:27:56.514932 29608 catalog_manager.cc:5495] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 468cc626e28c473ebaf71388ed6b1e69 (127.12.78.2). New cstate: current_term: 1 leader_uuid: "468cc626e28c473ebaf71388ed6b1e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "468cc626e28c473ebaf71388ed6b1e69" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 39647 } health_report { overall_health: HEALTHY } } }
I20260809 20:27:56.516057 29952 leader_election.cc:302] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 309996c7d4e24785a7c198edf339f563; no voters: 468cc626e28c473ebaf71388ed6b1e69, 670837917ff8439386f4315d00fc8004
I20260809 20:27:56.516686 30088 raft_consensus.cc:2726] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 20:27:56.633723 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.628222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.12.78.2:39647)
0809 20:27:56.628399 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:56.628403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:56.628798 (+ 395us) server_negotiation.cc:407] Connection header received
0809 20:27:56.628978 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:56.628982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:56.629060 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:56.629197 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:27:56.629543 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.630237 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.631662 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.632038 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.632134 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.632667 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:27:56.633175 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:27:56.633287 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:56.633401 (+ 114us) server_negotiation.cc:299] Negotiation successful
0809 20:27:56.633465 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 20:27:56.641286 29752 raft_consensus.cc:1261] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004 [term 1 FOLLOWER]: Refusing update from remote peer 468cc626e28c473ebaf71388ed6b1e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:27:56.641336 30021 raft_consensus.cc:1261] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [term 1 FOLLOWER]: Refusing update from remote peer 468cc626e28c473ebaf71388ed6b1e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:27:56.642195 30095 consensus_queue.cc:1035] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309996c7d4e24785a7c198edf339f563" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:27:56.642529 30090 consensus_queue.cc:1035] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "670837917ff8439386f4315d00fc8004" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 34943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:27:56.654039 30100 mvcc.cc:205] Tried to move back new op lower bound from 7316714605113065472 to 7316714604587806720. Current Snapshot: MvccSnapshot[applied={T|T < 7316714605113065472}]
I20260809 20:27:56.669791 29607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:27:56.670120 29607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:27:56.673837 29607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Sending DeleteTablet for 3 replicas of tablet aec4ef95717049c2b8b681f3bcec21d1
I20260809 20:27:56.675211 30000 tablet_service.cc:1514] Processing DeleteTablet for tablet aec4ef95717049c2b8b681f3bcec21d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:27:56 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:37412
I20260809 20:27:56.675318 29732 tablet_service.cc:1514] Processing DeleteTablet for tablet aec4ef95717049c2b8b681f3bcec21d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:27:56 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:59520
I20260809 20:27:56.675601 29866 tablet_service.cc:1514] Processing DeleteTablet for tablet aec4ef95717049c2b8b681f3bcec21d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:27:56 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:34998
I20260809 20:27:56.676646 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 29647
I20260809 20:27:56.688020 30112 tablet_replica.cc:330] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563: stopping tablet replica
I20260809 20:27:56.689352 30114 tablet_replica.cc:330] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69: stopping tablet replica
I20260809 20:27:56.689607 30112 raft_consensus.cc:2229] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:27:56.689723 30113 tablet_replica.cc:330] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004: stopping tablet replica
I20260809 20:27:56.690191 30112 raft_consensus.cc:2258] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:27:56.690181 30113 raft_consensus.cc:2229] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:27:56.690747 30113 raft_consensus.cc:2258] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:27:56.692528 30114 raft_consensus.cc:2229] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:27:56.692862 30113 ts_tablet_manager.cc:1905] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:27:56.693472 30114 raft_consensus.cc:2258] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:27:56.693802 30112 ts_tablet_manager.cc:1905] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:27:56.697649 30114 ts_tablet_manager.cc:1905] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:27:56.700649 30113 ts_tablet_manager.cc:1918] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:27:56.700845 30113 log.cc:1196] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/aec4ef95717049c2b8b681f3bcec21d1
I20260809 20:27:56.701443 30113 ts_tablet_manager.cc:1939] T aec4ef95717049c2b8b681f3bcec21d1 P 670837917ff8439386f4315d00fc8004: Deleting consensus metadata
I20260809 20:27:56.702272 30112 ts_tablet_manager.cc:1918] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:27:56.702455 30112 log.cc:1196] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/aec4ef95717049c2b8b681f3bcec21d1
I20260809 20:27:56.703002 30112 ts_tablet_manager.cc:1939] T aec4ef95717049c2b8b681f3bcec21d1 P 309996c7d4e24785a7c198edf339f563: Deleting consensus metadata
I20260809 20:27:56.703719 29594 catalog_manager.cc:4841] TS 670837917ff8439386f4315d00fc8004 (127.12.78.1:34943): tablet aec4ef95717049c2b8b681f3bcec21d1 (table test-table [id=3251ca1bed6e4e30b9b3622da2d7588b]) successfully deleted
I20260809 20:27:56.705148 29595 catalog_manager.cc:4841] TS 309996c7d4e24785a7c198edf339f563 (127.12.78.3:37971): tablet aec4ef95717049c2b8b681f3bcec21d1 (table test-table [id=3251ca1bed6e4e30b9b3622da2d7588b]) successfully deleted
I20260809 20:27:56.709394 30114 ts_tablet_manager.cc:1918] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:27:56.709635 30114 log.cc:1196] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/aec4ef95717049c2b8b681f3bcec21d1
I20260809 20:27:56.710160 30114 ts_tablet_manager.cc:1939] T aec4ef95717049c2b8b681f3bcec21d1 P 468cc626e28c473ebaf71388ed6b1e69: Deleting consensus metadata
I20260809 20:27:56.712069 29594 catalog_manager.cc:4841] TS 468cc626e28c473ebaf71388ed6b1e69 (127.12.78.2:39647): tablet aec4ef95717049c2b8b681f3bcec21d1 (table test-table [id=3251ca1bed6e4e30b9b3622da2d7588b]) successfully deleted
I20260809 20:27:56.714012 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.678097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.12.78.1:34943)
0809 20:27:56.678287 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:56.678294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:56.688528 (+ 10234us) server_negotiation.cc:407] Connection header received
0809 20:27:56.688816 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:56.688822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:56.688947 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:56.689072 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:56.692155 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.693307 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.695852 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.696563 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.696701 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.706046 (+ 9345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:56.706063 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:56.706083 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:56.706146 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:56.710397 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:56.711466 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:56.711476 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:56.711480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:56.711655 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:56.712148 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:56.712154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:56.712158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:56.712670 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:56.712984 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:56.713258 (+ 274us) server_negotiation.cc:299] Negotiation successful
0809 20:27:56.713503 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 20:27:56.714694 29793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:56.809741 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 29647
I20260809 20:27:56.836853 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 29802
I20260809 20:27:56.853724 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.837729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.12.78.2:39647)
0809 20:27:56.837925 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:56.837932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:56.838016 (+ 84us) server_negotiation.cc:407] Connection header received
0809 20:27:56.838196 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:56.838203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:56.838315 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:56.838430 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:56.839830 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.841120 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.842438 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.843065 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:56.843203 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:56.846047 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:56.846066 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:56.846095 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:56.846149 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:56.850537 (+ 4388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:56.851355 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:56.851366 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:56.851370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:56.851543 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:56.851913 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:56.851919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:56.851922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:56.852442 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:56.852791 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:56.853081 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 20:27:56.853344 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 20:27:56.854673 29926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:56.973510 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 29802
I20260809 20:27:56.995328 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 29934
I20260809 20:27:57.011209 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:27:56.996166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.12.78.3:37971)
0809 20:27:56.996681 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0809 20:27:56.996691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:27:56.996706 (+ 15us) server_negotiation.cc:407] Connection header received
0809 20:27:56.996760 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:27:56.996764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:27:56.996843 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:27:56.996964 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:27:56.998513 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:56.999282 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:57.000992 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:27:57.001467 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:27:57.001623 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:27:57.004702 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:27:57.004719 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:27:57.004736 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:27:57.004774 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:27:57.007794 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:57.008844 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:57.008861 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:57.008863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:57.009016 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:27:57.009459 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:27:57.009465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:27:57.009470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:27:57.009928 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:27:57.010275 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0809 20:27:57.010574 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 20:27:57.010819 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":235,"threads_started":1}
I20260809 20:27:57.011879 30061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:57.103312 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 29934
I20260809 20:27:57.124814 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 29576
I20260809 20:27:57.126045 29640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:27:57.241487 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 29576
I20260809 20:28:17.267115 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:42277
--webserver_interface=127.12.78.62
--webserver_port=34291
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:42277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:17.628454 30121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:17.628829 30121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:17.629010 30121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:17.636474 30121 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 20:28:17.636668 30121 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 20:28:17.636766 30121 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 20:28:17.636878 30121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:28:17.636971 30121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:17.637082 30121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:17.637169 30121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:28:17.637250 30121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:28:17.647806 30121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:42277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:42277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=34291
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:17.649245 30121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:17.650658 30121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:17.663225 30126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:17.663290 30127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:17.663321 30129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:17.663959 30121 server_base.cc:1034] running on GCE node
I20260809 20:28:17.664876 30121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:17.666322 30121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:17.667568 30121 hybrid_clock.cc:648] HybridClock initialized: now 1786307297667493 us; error 50 us; skew 500 ppm
I20260809 20:28:17.667903 30121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:17 dist-test-slave-rnrw krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307297, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:17.671223 30121 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:28:17.672945 30121 webserver.cc:460] Webserver started at http://127.12.78.62:34291/ using document root <none> and password file <none>
I20260809 20:28:17.673761 30121 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:17.673899 30121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:17.679725 30121 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 20:28:17.683068 30136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:17.684862 30121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:28:17.685146 30121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "a9041503701d4e079f66889a0a249e58"
format_stamp: "Formatted at 2026-08-09 20:27:31 on dist-test-slave-rnrw"
server_key: "99391ddb2d1e8d28ff12284d6f8f0fd2"
server_key_iv: "6a1ac375fc5c777e617237ffcfd84307"
server_key_version: "encryptionkey@0"
I20260809 20:28:17.686347 30121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:17.703390 30121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:17.708884 30121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:17.709272 30121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:17.730212 30121 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:42277
I20260809 20:28:17.730232 30189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:42277 every 8 connection(s)
I20260809 20:28:17.731967 30121 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:28:17.739522 30190 sys_catalog.cc:263] Verifying existing consensus state
I20260809 20:28:17.739686 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 30121
I20260809 20:28:17.740851 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:34943
--local_ip_for_outbound_sockets=127.12.78.1
--tserver_master_addrs=127.12.78.62:42277
--webserver_port=41487
--webserver_interface=127.12.78.1
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:28:17.744855 30190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Bootstrap starting.
I20260809 20:28:17.768335 30190 log.cc:824] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:17.787227 30190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 20:28:17.787892 30190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Bootstrap complete.
I20260809 20:28:17.799043 30190 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } }
I20260809 20:28:17.800069 30190 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9041503701d4e079f66889a0a249e58, State: Initialized, Role: FOLLOWER
I20260809 20:28:17.800835 30190 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } }
I20260809 20:28:17.801101 30190 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:28:17.801246 30190 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:28:17.801474 30190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 2 FOLLOWER]: Advancing to term 3
I20260809 20:28:17.805987 30190 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } }
I20260809 20:28:17.806533 30190 leader_election.cc:302] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9041503701d4e079f66889a0a249e58; no voters:
I20260809 20:28:17.807340 30190 leader_election.cc:288] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 20:28:17.807528 30194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 20:28:17.808866 30194 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [term 3 LEADER]: Becoming Leader. State: Replica: a9041503701d4e079f66889a0a249e58, State: Running, Role: LEADER
I20260809 20:28:17.809604 30194 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } }
I20260809 20:28:17.809865 30190 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:28:17.812304 30196 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9041503701d4e079f66889a0a249e58. Latest consensus state: current_term: 3 leader_uuid: "a9041503701d4e079f66889a0a249e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } } }
I20260809 20:28:17.812738 30196 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:17.812508 30195 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a9041503701d4e079f66889a0a249e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9041503701d4e079f66889a0a249e58" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42277 } } }
I20260809 20:28:17.812950 30195 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:17.824170 30201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:28:17.836858 30201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:28:17.839164 30201 catalog_manager.cc:1259] Loaded cluster ID: f0b8b1bd09644b7aa8dc00c67995a08d
I20260809 20:28:17.839392 30201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:28:17.845552 30201 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:28:17.848318 30201 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Loaded TSK: 0
I20260809 20:28:17.848524 30201 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9041503701d4e079f66889a0a249e58: Loaded TSK: 1
F20260809 20:28:17.849630 30201 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa9a7db59ad google::LogMessage::Fail() at ??:0
@ 0x7fa9a7db888a google::LogMessage::SendToLog() at ??:0
@ 0x7fa9a7db54de google::LogMessage::Flush() at ??:0
@ 0x7fa9a7db6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa9abb197d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa9abb0873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa9bd7e81c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa9bd7e7783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa9bd8732a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fa9bd8730c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa9bd8bd5e9 std::function<>::operator()() at ??:0
@ 0x7fa9bd7ebcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa9bd7e9c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa9bd871610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa9bd871439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa9aaa01345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa9aaa0491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa9aaa04799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa9aa9dfeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa9adcb96db start_thread at ??:0
@ 0x7fa9a6c4f61f clone at ??:0
W20260809 20:28:18.171072 30192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:18.171375 30192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:18.171535 30192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:18.179175 30192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:18.179301 30192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:18.179510 30192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:28:18.190452 30192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:34943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=41487
--tserver_master_addrs=127.12.78.62:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:18.191871 30192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:18.193455 30192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:18.209151 30237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:18.209173 30238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:18.209127 30240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:18.210182 30192 server_base.cc:1034] running on GCE node
I20260809 20:28:18.210868 30192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:18.211890 30192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:18.213145 30192 hybrid_clock.cc:648] HybridClock initialized: now 1786307298213079 us; error 72 us; skew 500 ppm
I20260809 20:28:18.213657 30192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:18 dist-test-slave-rnrw krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307298, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:18.217368 30192 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:28:18.218983 30192 webserver.cc:460] Webserver started at http://127.12.78.1:41487/ using document root <none> and password file <none>
I20260809 20:28:18.219748 30192 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:18.219906 30192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:18.226050 30192 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 20:28:18.229110 30249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:18.230576 30192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 20:28:18.230791 30192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "670837917ff8439386f4315d00fc8004"
format_stamp: "Formatted at 2026-08-09 20:27:31 on dist-test-slave-rnrw"
server_key: "40c9e00144d1e680b42d1c26fbb1db1f"
server_key_iv: "5dfc7bf660a38693fbd3ba48015167e6"
server_key_version: "encryptionkey@0"
I20260809 20:28:18.231854 30192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:18.253310 30192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:18.258105 30192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:18.258410 30192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:18.259601 30192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:18.261168 30192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:18.261296 30192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:18.261445 30192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:18.261544 30192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:18.298988 30192 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:34943
I20260809 20:28:18.299057 30363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:34943 every 8 connection(s)
I20260809 20:28:18.300478 30192 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:28:18.305768 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 30192
I20260809 20:28:18.306253 30367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:18.304025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:45655)
0809 20:28:18.304973 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:18.305232 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":326,"threads_started":1}
I20260809 20:28:18.306506 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:39647
--local_ip_for_outbound_sockets=127.12.78.2
--tserver_master_addrs=127.12.78.62:42277
--webserver_port=35361
--webserver_interface=127.12.78.2
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:18.306983 30365 heartbeater.cc:644] Failed to heartbeat to 127.12.78.62:42277 (0 consecutive failures): Network error: Failed to ping master at 127.12.78.62:42277: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
I20260809 20:28:18.307914 30367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:18.307551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:38183)
0809 20:28:18.307719 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:18.307799 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260809 20:28:18.308825 30367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:18.308456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:51991)
0809 20:28:18.308627 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:18.308712 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 20:28:18.309073 30365 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 20:28:18.664268 30368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:18.664621 30368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:18.664744 30368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:18.672179 30368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:18.672358 30368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:18.672516 30368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:28:18.682664 30368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:39647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=35361
--tserver_master_addrs=127.12.78.62:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:18.683990 30368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:18.685370 30368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:18.699014 30373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:18.698975 30374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:18.699044 30376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:18.700416 30368 server_base.cc:1034] running on GCE node
I20260809 20:28:18.701176 30368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:18.702275 30368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:18.703588 30368 hybrid_clock.cc:648] HybridClock initialized: now 1786307298703532 us; error 43 us; skew 500 ppm
I20260809 20:28:18.703994 30368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:18 dist-test-slave-rnrw krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307298, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:18.707923 30368 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:28:18.709764 30368 webserver.cc:460] Webserver started at http://127.12.78.2:35361/ using document root <none> and password file <none>
I20260809 20:28:18.710569 30368 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:18.710736 30368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:18.717032 30368 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 20:28:18.720227 30383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:18.721729 30368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:28:18.721940 30368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "468cc626e28c473ebaf71388ed6b1e69"
format_stamp: "Formatted at 2026-08-09 20:27:32 on dist-test-slave-rnrw"
server_key: "1c607ceec687b50da5002c8943be9c7e"
server_key_iv: "3b827a162327df7e574ca83e0a675c19"
server_key_version: "encryptionkey@0"
I20260809 20:28:18.723100 30368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:18.753252 30368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:18.758348 30368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:18.758701 30368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:18.759881 30368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:18.761458 30368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:18.761603 30368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:18.761754 30368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:18.761852 30368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:18.806421 30368 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:39647
I20260809 20:28:18.806573 30499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:39647 every 8 connection(s)
I20260809 20:28:18.808141 30368 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:28:18.813453 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:18.811452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:43007)
0809 20:28:18.812234 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:18.812471 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":320,"threads_started":1}
W20260809 20:28:18.814241 30500 heartbeater.cc:644] Failed to heartbeat to 127.12.78.62:42277 (0 consecutive failures): Network error: Failed to ping master at 127.12.78.62:42277: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
I20260809 20:28:18.815219 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:18.814838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:57543)
0809 20:28:18.815011 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:18.815099 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 20:28:18.816131 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:18.815771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:52391)
0809 20:28:18.815921 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:18.816008 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 20:28:18.816397 30500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 20:28:18.817415 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 30368
I20260809 20:28:18.817999 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:37971
--local_ip_for_outbound_sockets=127.12.78.3
--tserver_master_addrs=127.12.78.62:42277
--webserver_port=33553
--webserver_interface=127.12.78.3
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:19.168298 30503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:19.168682 30503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:19.168802 30503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:19.177251 30503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:19.177379 30503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:19.177498 30503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:28:19.187745 30503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:37971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=33553
--tserver_master_addrs=127.12.78.62:42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:19.189136 30503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:19.190569 30503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:19.205909 30508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:19.205984 30509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:19.205932 30511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:19.206774 30503 server_base.cc:1034] running on GCE node
I20260809 20:28:19.207507 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:19.208689 30503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:19.209944 30503 hybrid_clock.cc:648] HybridClock initialized: now 1786307299209863 us; error 66 us; skew 500 ppm
I20260809 20:28:19.210301 30503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:19 dist-test-slave-rnrw krb5kdc[29007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307299, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:19.213685 30503 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:28:19.215422 30503 webserver.cc:460] Webserver started at http://127.12.78.3:33553/ using document root <none> and password file <none>
I20260809 20:28:19.216169 30503 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:19.216284 30503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:19.222146 30503 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 20:28:19.225157 30518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:19.226696 30503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 20:28:19.226878 30503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "309996c7d4e24785a7c198edf339f563"
format_stamp: "Formatted at 2026-08-09 20:27:32 on dist-test-slave-rnrw"
server_key: "42c06bad7fdde7161dc1beaa5cc9fb9f"
server_key_iv: "fa84e20a1680c089dbc2542df14a30d7"
server_key_version: "encryptionkey@0"
I20260809 20:28:19.227929 30503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:19.243552 30503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:19.248185 30503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:19.248492 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:19.249646 30503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:19.251134 30503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:19.251263 30503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:19.251399 30503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:19.251477 30503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:19.287541 30503 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:37971
I20260809 20:28:19.287640 30632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:37971 every 8 connection(s)
I20260809 20:28:19.289191 30503 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:28:19.294763 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:19.292419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:45263)
0809 20:28:19.293298 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:19.293613 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":319,"threads_started":1}
W20260809 20:28:19.295571 30634 heartbeater.cc:644] Failed to heartbeat to 127.12.78.62:42277 (0 consecutive failures): Network error: Failed to ping master at 127.12.78.62:42277: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
I20260809 20:28:19.296756 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:19.296250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:38255)
0809 20:28:19.296430 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:19.296630 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 20:28:19.297670 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:19.297289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:58551)
0809 20:28:19.297430 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:19.297555 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 20:28:19.297946 30634 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 20:28:19.299274 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 30503
I20260809 20:28:19.310684 30638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:19.309743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:57589)
0809 20:28:19.310259 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:19.310357 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
I20260809 20:28:19.817785 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:19.816993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:55655)
0809 20:28:19.817452 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:19.817544 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":222,"threads_started":1}
I20260809 20:28:20.299705 30640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:20.298679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:36609)
0809 20:28:20.299262 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:20.299413 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":256,"threads_started":1}
I20260809 20:28:20.312584 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:20.311774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:40309)
0809 20:28:20.312277 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:20.312358 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":266,"threads_started":1}
I20260809 20:28:20.819480 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:20.818735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:46745)
0809 20:28:20.819223 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:20.819309 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":230,"threads_started":1}
I20260809 20:28:21.301589 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:21.300796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:49859)
0809 20:28:21.301281 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:21.301370 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":222,"threads_started":1}
I20260809 20:28:21.314271 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:21.313505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:45369)
0809 20:28:21.313989 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:21.314081 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":246,"threads_started":1}
I20260809 20:28:21.821281 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:21.820394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:54419)
0809 20:28:21.821015 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:21.821099 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":413,"threads_started":1}
I20260809 20:28:22.303190 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:22.302476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:55875)
0809 20:28:22.302914 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:22.302997 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":225,"threads_started":1}
I20260809 20:28:22.316071 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:22.315227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:60711)
0809 20:28:22.315794 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:22.315881 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":334,"threads_started":1}
I20260809 20:28:22.822918 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:22.822200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:59291)
0809 20:28:22.822666 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:22.822749 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":232,"threads_started":1}
I20260809 20:28:23.305186 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:23.304194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:46869)
0809 20:28:23.304868 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:23.304969 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":284,"threads_started":1}
I20260809 20:28:23.317896 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:23.317119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:40861)
0809 20:28:23.317625 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:23.317711 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":242,"threads_started":1}
I20260809 20:28:23.824806 30651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:23.823926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:38373)
0809 20:28:23.824429 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:23.824622 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":259,"threads_started":1}
I20260809 20:28:24.306977 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:24.306170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:51693)
0809 20:28:24.306700 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:24.306805 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":280,"threads_started":1}
I20260809 20:28:24.319811 30653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:24.318926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:46927)
0809 20:28:24.319529 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:24.319640 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":336,"threads_started":1}
I20260809 20:28:24.826640 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:24.825858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:48823)
0809 20:28:24.826358 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:24.826455 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":228,"threads_started":1}
I20260809 20:28:25.308755 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:25.307879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:36965)
0809 20:28:25.308364 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:25.308460 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":247,"threads_started":1}
I20260809 20:28:25.321766 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:25.320860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:49861)
0809 20:28:25.321462 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:25.321551 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":318,"threads_started":1}
I20260809 20:28:25.828315 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:25.827479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:39835)
0809 20:28:25.827960 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:25.828099 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":201,"threads_started":1}
I20260809 20:28:26.310623 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:26.309771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:55971)
0809 20:28:26.310305 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:26.310389 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":255,"threads_started":1}
I20260809 20:28:26.323576 30659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:26.322782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:34249)
0809 20:28:26.323309 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:26.323399 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":294,"threads_started":1}
I20260809 20:28:26.830099 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:26.829255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:46995)
0809 20:28:26.829741 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:26.829871 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":234,"threads_started":1}
I20260809 20:28:27.312456 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:27.311692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:52325)
0809 20:28:27.312203 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:27.312289 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":262,"threads_started":1}
I20260809 20:28:27.325322 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:27.324503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:36509)
0809 20:28:27.325064 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:27.325146 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":237,"threads_started":1}
I20260809 20:28:27.832207 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:27.831068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:44425)
0809 20:28:27.831937 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:27.832023 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":580,"threads_started":1}
I20260809 20:28:28.314761 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:28.313730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:53409)
0809 20:28:28.314499 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:28.314582 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":354,"threads_started":1}
I20260809 20:28:28.327147 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:28.326265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:56839)
0809 20:28:28.326873 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:28.326958 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":378,"threads_started":1}
I20260809 20:28:28.834252 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:28.833409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:59935)
0809 20:28:28.833967 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:28.834049 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":324,"threads_started":1}
I20260809 20:28:29.316847 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:29.315866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:50113)
0809 20:28:29.316501 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:29.316649 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":275,"threads_started":1}
I20260809 20:28:29.329128 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:29.328238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:49665)
0809 20:28:29.328829 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:29.328936 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":349,"threads_started":1}
I20260809 20:28:29.836028 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:29.835270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:58609)
0809 20:28:29.835720 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:29.835824 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":204,"threads_started":1}
I20260809 20:28:30.318688 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:30.317855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:58359)
0809 20:28:30.318326 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:30.318451 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":234,"threads_started":1}
I20260809 20:28:30.330881 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:30.330147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:57843)
0809 20:28:30.330621 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:30.330710 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260809 20:28:30.837771 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:30.837047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:52115)
0809 20:28:30.837494 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:30.837612 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":229,"threads_started":1}
I20260809 20:28:31.320463 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:31.319726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:47583)
0809 20:28:31.320198 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:31.320284 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":247,"threads_started":1}
I20260809 20:28:31.332872 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:31.331903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:36031)
0809 20:28:31.332434 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:31.332535 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":263,"threads_started":1}
I20260809 20:28:31.839547 30675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:31.838684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:50413)
0809 20:28:31.839231 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:31.839353 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":232,"threads_started":1}
I20260809 20:28:32.322315 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:32.321475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:43743)
0809 20:28:32.322028 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:32.322126 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":253,"threads_started":1}
I20260809 20:28:32.334635 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:32.333804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:49245)
0809 20:28:32.334341 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:32.334429 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":303,"threads_started":1}
I20260809 20:28:32.841243 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:32.840436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:56895)
0809 20:28:32.840964 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:32.841048 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":296,"threads_started":1}
I20260809 20:28:33.323902 30679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:33.323183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:44661)
0809 20:28:33.323633 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:33.323715 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":226,"threads_started":1}
I20260809 20:28:33.336728 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:33.335826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:34171)
0809 20:28:33.336322 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:33.336455 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":265,"threads_started":1}
I20260809 20:28:33.843155 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:33.842375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.2:60209)
0809 20:28:33.842837 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:33.842956 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":231,"threads_started":1}
I20260809 20:28:34.301630 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 30192
I20260809 20:28:34.325847 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:34.325045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.3:41711)
0809 20:28:34.325572 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:34.325671 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":286,"threads_started":1}
I20260809 20:28:34.326880 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:34.302913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.12.78.1:34943)
0809 20:28:34.303833 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:34.303856 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:34.303907 (+ 51us) server_negotiation.cc:407] Connection header received
0809 20:28:34.305738 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:34.305748 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:34.306177 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:34.306473 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:34.307957 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:34.309263 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:34.310626 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:34.311307 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:34.311453 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:34.319681 (+ 8228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:34.319691 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:34.319707 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:34.319756 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:34.322988 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:34.323732 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:34.323740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:34.323747 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:34.323930 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:34.324400 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:34.324404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:34.324407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:34.325102 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:34.325472 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:34.326004 (+ 532us) server_negotiation.cc:299] Negotiation successful
0809 20:28:34.326469 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":529,"threads_started":1}
I20260809 20:28:34.328425 30359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:34.414431 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 30192
I20260809 20:28:34.414587 30686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:34.413374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42277 (local address 127.12.78.1:42925)
0809 20:28:34.413999 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:34.414200 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.62:42277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":290,"threads_started":1}
I20260809 20:28:34.436394 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 30368
I20260809 20:28:34.455046 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:34.437416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.12.78.2:39647)
0809 20:28:34.437986 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:34.438001 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:34.438038 (+ 37us) server_negotiation.cc:407] Connection header received
0809 20:28:34.439634 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:34.439645 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:34.440085 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:34.440474 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:34.442169 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:34.443446 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:34.444861 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:34.445574 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:34.445801 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:34.448644 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:34.448658 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:34.448682 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:34.448743 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:34.451591 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:34.452329 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:34.452344 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:34.452352 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:34.452604 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:34.452994 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:34.453000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:34.453004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:34.453530 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:34.453875 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:34.454315 (+ 440us) server_negotiation.cc:299] Negotiation successful
0809 20:28:34.454753 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":259,"threads_started":1}
I20260809 20:28:34.456450 30495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:34.541887 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 30368
I20260809 20:28:34.562465 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 30503
I20260809 20:28:34.581036 30689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:34.563556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.12.78.3:37971)
0809 20:28:34.564218 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:34.564232 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:34.564276 (+ 44us) server_negotiation.cc:407] Connection header received
0809 20:28:34.565972 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:34.565984 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:34.566431 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:34.566721 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:34.568221 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:34.569607 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:34.570932 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:34.571638 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:34.571782 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:34.574490 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:34.574498 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:34.574514 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:34.574564 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:34.577736 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:34.578426 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:34.578432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:34.578439 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:34.578614 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:34.579049 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:34.579052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:34.579054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:34.579560 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:34.579771 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:34.580229 (+ 458us) server_negotiation.cc:299] Negotiation successful
0809 20:28:34.580732 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":310,"threads_started":1}
I20260809 20:28:34.582512 30628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:34.665772 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 30503
2026-08-09T20:28:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65367 ms)
[----------] 30 tests from SecurityITest (312826 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:28:34 dist-test-slave-rnrw krb5kdc[30693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:28:34 dist-test-slave-rnrw krb5kdc[30693](info): set up 2 sockets
Aug 09 20:28:34 dist-test-slave-rnrw krb5kdc[30693](info): commencing operation
krb5kdc: starting...
W20260809 20:28:35.905237 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:28:36 dist-test-slave-rnrw krb5kdc[30693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:28:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:28:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:28:36.204829 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:46457
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:36.556100 30706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:36.556384 30706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:36.556536 30706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:36.563807 30706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:28:36.563913 30706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:36.564018 30706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:36.564116 30706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:28:36.564200 30706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:28:36.574204 30706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:46457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:36.575544 30706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:36.576956 30706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:36.588950 30711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:36.588963 30714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:36.589649 30712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:36.589953 30706 server_base.cc:1034] running on GCE node
I20260809 20:28:36.590742 30706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:36.592160 30706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:36.593400 30706 hybrid_clock.cc:648] HybridClock initialized: now 1786307316593355 us; error 51 us; skew 500 ppm
I20260809 20:28:36.593760 30706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:36 dist-test-slave-rnrw krb5kdc[30693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307316, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:36.596972 30706 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:28:36.598508 30706 webserver.cc:460] Webserver started at http://127.12.78.62:38747/ using document root <none> and password file <none>
I20260809 20:28:36.599232 30706 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:36.599385 30706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:36.599702 30706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:36.602716 30706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "1bfceda790084b41aab898e09d209aa5"
format_stamp: "Formatted at 2026-08-09 20:28:36 on dist-test-slave-rnrw"
server_key: "c5181cb392d30831df608e6c9b2383c4"
server_key_iv: "5fe7a731bc6ecbdd3abbb1054c550171"
server_key_version: "encryptionkey@0"
I20260809 20:28:36.603575 30706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "1bfceda790084b41aab898e09d209aa5"
format_stamp: "Formatted at 2026-08-09 20:28:36 on dist-test-slave-rnrw"
server_key: "c5181cb392d30831df608e6c9b2383c4"
server_key_iv: "5fe7a731bc6ecbdd3abbb1054c550171"
server_key_version: "encryptionkey@0"
I20260809 20:28:36.609041 30706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 20:28:36.612758 30721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:36.614116 30706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:28:36.614331 30706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "1bfceda790084b41aab898e09d209aa5"
format_stamp: "Formatted at 2026-08-09 20:28:36 on dist-test-slave-rnrw"
server_key: "c5181cb392d30831df608e6c9b2383c4"
server_key_iv: "5fe7a731bc6ecbdd3abbb1054c550171"
server_key_version: "encryptionkey@0"
I20260809 20:28:36.614575 30706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:36.639221 30706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:36.643981 30706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:36.644282 30706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:36.663512 30706 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:46457
I20260809 20:28:36.663547 30773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:46457 every 8 connection(s)
I20260809 20:28:36.665066 30706 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:28:36.668866 30774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:36.674333 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 30706
I20260809 20:28:36.674572 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:28:36.675254 12600 external_mini_cluster.cc:1428] Setting key ef323699b8f9221bf54aa446b109a9ee
I20260809 20:28:36.679780 30774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5: Bootstrap starting.
I20260809 20:28:36.683568 30774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:36.684819 30774 log.cc:824] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:36.688048 30774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5: No bootstrap required, opened a new log
Aug 09 20:28:36 dist-test-slave-rnrw krb5kdc[30693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:28:36.698902 30774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfceda790084b41aab898e09d209aa5" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46457 } }
I20260809 20:28:36.699374 30774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:36.699491 30774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfceda790084b41aab898e09d209aa5, State: Initialized, Role: FOLLOWER
I20260809 20:28:36.700147 30774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfceda790084b41aab898e09d209aa5" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46457 } }
I20260809 20:28:36.700431 30774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:28:36.700630 30774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:28:36.700831 30774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:36.700877 30777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:36.677321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.12.78.62:46457)
0809 20:28:36.678154 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:36.678165 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:36.678206 (+ 41us) server_negotiation.cc:407] Connection header received
0809 20:28:36.679868 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:36.679891 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:36.680323 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:36.680753 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:36.682645 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:36.684024 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:36.685927 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:36.686634 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:36.686768 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:36.691874 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:36.691886 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:36.691910 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:36.691960 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:36.696084 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:36.696904 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:36.696913 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:36.696922 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:36.697173 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:36.697548 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:36.697553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:36.697557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:36.698304 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:36.698593 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:36.699002 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 20:28:36.699458 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":404,"threads_started":1}
I20260809 20:28:36.703276 30774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfceda790084b41aab898e09d209aa5" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46457 } }
I20260809 20:28:36.703833 30774 leader_election.cc:302] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bfceda790084b41aab898e09d209aa5; no voters:
I20260809 20:28:36.704391 30774 leader_election.cc:288] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:28:36.704527 30779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:28:36.705274 30779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [term 1 LEADER]: Becoming Leader. State: Replica: 1bfceda790084b41aab898e09d209aa5, State: Running, Role: LEADER
I20260809 20:28:36.705796 30779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfceda790084b41aab898e09d209aa5" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46457 } }
I20260809 20:28:36.706630 30774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:28:36.708885 30780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bfceda790084b41aab898e09d209aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfceda790084b41aab898e09d209aa5" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46457 } } }
I20260809 20:28:36.709363 30780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:36.710098 30782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bfceda790084b41aab898e09d209aa5. Latest consensus state: current_term: 1 leader_uuid: "1bfceda790084b41aab898e09d209aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfceda790084b41aab898e09d209aa5" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46457 } } }
I20260809 20:28:36.710513 30782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:36.713713 30787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:28:36.718802 30787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:28:36.728824 30787 catalog_manager.cc:1347] Generated new cluster ID: 80581d93653a4f9f8858daf67e238e92
I20260809 20:28:36.728983 30787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:28:36.762423 30787 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:28:36.764411 30787 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:28:36.782982 30787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5: Generated new TSK 0
I20260809 20:28:36.784044 30787 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:28:36.856302 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:46457
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:37.209143 30802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:37.209466 30802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:37.209635 30802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:37.217590 30802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:37.217741 30802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:37.217967 30802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:28:37.228204 30802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:46457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:37.229617 30802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:37.230943 30802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:37.244789 30810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:37.244860 30808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:37.244864 30807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:37.245942 30802 server_base.cc:1034] running on GCE node
I20260809 20:28:37.246639 30802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:37.247718 30802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:37.248924 30802 hybrid_clock.cc:648] HybridClock initialized: now 1786307317248879 us; error 36 us; skew 500 ppm
I20260809 20:28:37.249276 30802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:37 dist-test-slave-rnrw krb5kdc[30693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:37.252666 30802 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:28:37.254148 30802 webserver.cc:460] Webserver started at http://127.12.78.1:36553/ using document root <none> and password file <none>
I20260809 20:28:37.254813 30802 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:37.254918 30802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:37.255230 30802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:37.258309 30802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "893d554182c645d180d0d187d6a0f62b"
format_stamp: "Formatted at 2026-08-09 20:28:37 on dist-test-slave-rnrw"
server_key: "fc6859efdf75c82e704dccf9dc35b936"
server_key_iv: "5aa91b89a98347254620d8991599cd23"
server_key_version: "encryptionkey@0"
I20260809 20:28:37.259068 30802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "893d554182c645d180d0d187d6a0f62b"
format_stamp: "Formatted at 2026-08-09 20:28:37 on dist-test-slave-rnrw"
server_key: "fc6859efdf75c82e704dccf9dc35b936"
server_key_iv: "5aa91b89a98347254620d8991599cd23"
server_key_version: "encryptionkey@0"
I20260809 20:28:37.264668 30802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 20:28:37.269040 30817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:37.270730 30802 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 20:28:37.270931 30802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "893d554182c645d180d0d187d6a0f62b"
format_stamp: "Formatted at 2026-08-09 20:28:37 on dist-test-slave-rnrw"
server_key: "fc6859efdf75c82e704dccf9dc35b936"
server_key_iv: "5aa91b89a98347254620d8991599cd23"
server_key_version: "encryptionkey@0"
I20260809 20:28:37.271168 30802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:37.288342 30802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:37.292709 30802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:37.293021 30802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:37.294039 30802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:37.295429 30802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:37.295536 30802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:37.295694 30802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:37.295768 30802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:37.330389 30802 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:40037
I20260809 20:28:37.330436 30930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:40037 every 8 connection(s)
I20260809 20:28:37.331686 30802 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:28:37.337896 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 30802
I20260809 20:28:37.338148 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:28:37.338811 12600 external_mini_cluster.cc:1428] Setting key d64273c5f55fe2045a67e6d3f61f931c
Aug 09 20:28:37 dist-test-slave-rnrw krb5kdc[30693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:28:37.353181 30933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:37.334296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:40785 (local address 127.12.78.62:46457)
0809 20:28:37.334781 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:37.334787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:37.335700 (+ 913us) server_negotiation.cc:407] Connection header received
0809 20:28:37.337591 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:37.337598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:37.337693 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:37.337828 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:37.339413 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:37.340144 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:37.341894 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:37.342645 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:37.342799 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:37.346207 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:37.346215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:37.346222 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:37.346267 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:37.349999 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:37.350767 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:37.350775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:37.350781 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:37.350934 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:37.351340 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:37.351347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:37.351353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:37.351751 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:37.351906 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:37.352739 (+ 833us) server_negotiation.cc:299] Negotiation successful
0809 20:28:37.352923 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":231,"threads_started":1}
I20260809 20:28:37.354786 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:37.334837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:46457 (local address 127.12.78.1:40785)
0809 20:28:37.335547 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:37.335584 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:37.337318 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:37.338040 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:37.338052 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:37.338649 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:37.339241 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:37.339258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:37.340312 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:37.340317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:37.341748 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:37.341760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:37.342864 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:37.342869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:37.343074 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:37.343779 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:28:37.343801 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:37.345984 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:37.350151 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:37.350159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:37.350171 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:37.350634 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:37.351038 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:37.351042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:37.351044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:37.351225 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:37.351885 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:37.351892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:37.352210 (+ 318us) client_negotiation.cc:769] Sending connection context
0809 20:28:37.352644 (+ 434us) client_negotiation.cc:240] Negotiation successful
0809 20:28:37.353249 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":289,"threads_started":1}
I20260809 20:28:37.356498 30931 heartbeater.cc:347] Connected to a master server at 127.12.78.62:46457
I20260809 20:28:37.356878 30931 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:37.357620 30931 heartbeater.cc:511] Master 127.12.78.62:46457 requested a full tablet report, sending...
I20260809 20:28:37.359751 30738 ts_manager.cc:194] Registered new tserver with Master: 893d554182c645d180d0d187d6a0f62b (127.12.78.1:40037)
I20260809 20:28:37.361325 30738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:40785
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:28:37.403477 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:46457
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:37.741358 30938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:37.741636 30938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:37.741818 30938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:37.748869 30938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:37.748975 30938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:37.749195 30938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:28:37.758749 30938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:46457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:37.760028 30938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:37.761346 30938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:37.775332 30945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:37.775312 30943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:37.775368 30947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:37.776129 30938 server_base.cc:1034] running on GCE node
I20260809 20:28:37.776844 30938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:37.777834 30938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:37.779054 30938 hybrid_clock.cc:648] HybridClock initialized: now 1786307317778983 us; error 58 us; skew 500 ppm
I20260809 20:28:37.779341 30938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:37 dist-test-slave-rnrw krb5kdc[30693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:37.782603 30938 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:28:37.784300 30938 webserver.cc:460] Webserver started at http://127.12.78.2:43503/ using document root <none> and password file <none>
I20260809 20:28:37.785087 30938 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:37.785241 30938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:37.785573 30938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:37.788584 30938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "6f956c089ad84f408976d5517488960b"
format_stamp: "Formatted at 2026-08-09 20:28:37 on dist-test-slave-rnrw"
server_key: "91729e5eba9421ee3fd6e475723d7ff0"
server_key_iv: "5f5d4bbb62cf465e9fdc440efe924800"
server_key_version: "encryptionkey@0"
I20260809 20:28:37.789371 30938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "6f956c089ad84f408976d5517488960b"
format_stamp: "Formatted at 2026-08-09 20:28:37 on dist-test-slave-rnrw"
server_key: "91729e5eba9421ee3fd6e475723d7ff0"
server_key_iv: "5f5d4bbb62cf465e9fdc440efe924800"
server_key_version: "encryptionkey@0"
I20260809 20:28:37.794576 30938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 20:28:37.798241 30954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:37.799561 30938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 20:28:37.799741 30938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "6f956c089ad84f408976d5517488960b"
format_stamp: "Formatted at 2026-08-09 20:28:37 on dist-test-slave-rnrw"
server_key: "91729e5eba9421ee3fd6e475723d7ff0"
server_key_iv: "5f5d4bbb62cf465e9fdc440efe924800"
server_key_version: "encryptionkey@0"
I20260809 20:28:37.799942 30938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:37.817546 30938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:37.822237 30938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:37.822539 30938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:37.823514 30938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:37.824919 30938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:37.825022 30938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:37.825186 30938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:37.825261 30938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:37.859306 30938 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:40879
I20260809 20:28:37.859354 31069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:40879 every 8 connection(s)
I20260809 20:28:37.860808 30938 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:28:37.862432 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 30938
I20260809 20:28:37.862641 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:28:37.863250 12600 external_mini_cluster.cc:1428] Setting key bb58b47490be0bc415fcce5f581755da
Aug 09 20:28:37 dist-test-slave-rnrw krb5kdc[30693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:28:37.883052 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:37.863652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:43245 (local address 127.12.78.62:46457)
0809 20:28:37.864183 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:37.864191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:37.865320 (+ 1129us) server_negotiation.cc:407] Connection header received
0809 20:28:37.867203 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:37.867212 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:37.867308 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:37.867449 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:37.869090 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:37.870264 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:37.871907 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:37.872745 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:37.872895 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:37.877245 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:37.877249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:37.877252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:37.877287 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:37.879865 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:37.880670 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:37.880678 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:37.880686 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:37.880824 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:37.881275 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:37.881284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:37.881286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:37.881564 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:37.881722 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:37.882432 (+ 710us) server_negotiation.cc:299] Negotiation successful
0809 20:28:37.882662 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":284,"threads_started":1}
I20260809 20:28:37.884059 31073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:37.864255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:46457 (local address 127.12.78.2:43245)
0809 20:28:37.865150 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:37.865198 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:37.866910 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:37.867658 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:37.867668 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:37.868239 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:37.868908 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:37.868927 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:37.870443 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:37.870447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:37.871731 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:37.871743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:37.872971 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:37.872977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:37.873193 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:37.873932 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:28:37.873960 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:37.876992 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:37.880013 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:37.880022 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:37.880039 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:37.880485 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:37.880945 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:37.880951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:37.880954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:37.881153 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:37.881697 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:37.881705 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:37.882062 (+ 357us) client_negotiation.cc:769] Sending connection context
0809 20:28:37.882345 (+ 283us) client_negotiation.cc:240] Negotiation successful
0809 20:28:37.882900 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":402,"threads_started":1}
I20260809 20:28:37.885632 31070 heartbeater.cc:347] Connected to a master server at 127.12.78.62:46457
I20260809 20:28:37.885944 31070 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:37.886686 31070 heartbeater.cc:511] Master 127.12.78.62:46457 requested a full tablet report, sending...
I20260809 20:28:37.888339 30738 ts_manager.cc:194] Registered new tserver with Master: 6f956c089ad84f408976d5517488960b (127.12.78.2:40879)
I20260809 20:28:37.889676 30738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:43245
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:28:37.930250 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:46457
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:38.268916 31076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:38.269223 31076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:38.269402 31076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:38.279345 31076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:38.279484 31076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:38.279708 31076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:28:38.291313 31076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:46457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:38.292901 31076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:38.294399 31076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:38.309693 31082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:38.309645 31084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:38.309617 31081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:38.310356 31076 server_base.cc:1034] running on GCE node
I20260809 20:28:38.311023 31076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:38.312001 31076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:38.313243 31076 hybrid_clock.cc:648] HybridClock initialized: now 1786307318313173 us; error 61 us; skew 500 ppm
I20260809 20:28:38.313617 31076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:38 dist-test-slave-rnrw krb5kdc[30693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307318, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:38.316977 31076 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:28:38.318814 31076 webserver.cc:460] Webserver started at http://127.12.78.3:44639/ using document root <none> and password file <none>
I20260809 20:28:38.319551 31076 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:38.319661 31076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:38.319936 31076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:38.322965 31076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "286dc9f07370473085e1897c8beabcc5"
format_stamp: "Formatted at 2026-08-09 20:28:38 on dist-test-slave-rnrw"
server_key: "091803aff42c5f7fb6c24a4e5751d9ab"
server_key_iv: "a84e21a5c9d3160c9e0a385d7369fc7e"
server_key_version: "encryptionkey@0"
I20260809 20:28:38.323709 31076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "286dc9f07370473085e1897c8beabcc5"
format_stamp: "Formatted at 2026-08-09 20:28:38 on dist-test-slave-rnrw"
server_key: "091803aff42c5f7fb6c24a4e5751d9ab"
server_key_iv: "a84e21a5c9d3160c9e0a385d7369fc7e"
server_key_version: "encryptionkey@0"
I20260809 20:28:38.329108 31076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 20:28:38.332829 31091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:38.334522 31076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:28:38.334745 31076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "286dc9f07370473085e1897c8beabcc5"
format_stamp: "Formatted at 2026-08-09 20:28:38 on dist-test-slave-rnrw"
server_key: "091803aff42c5f7fb6c24a4e5751d9ab"
server_key_iv: "a84e21a5c9d3160c9e0a385d7369fc7e"
server_key_version: "encryptionkey@0"
I20260809 20:28:38.334956 31076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:38.349416 31076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:38.353785 31076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:38.354130 31076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:38.355214 31076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:38.356755 31076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:38.356873 31076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:38.357044 31076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:38.357126 31076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:38.364300 30931 heartbeater.cc:503] Master 127.12.78.62:46457 was elected leader, sending a full tablet report...
I20260809 20:28:38.394107 31076 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:38611
I20260809 20:28:38.394168 31205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:38611 every 8 connection(s)
I20260809 20:28:38.395589 31076 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:28:38.399806 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 31076
I20260809 20:28:38.400012 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:28:38.400404 12600 external_mini_cluster.cc:1428] Setting key 23322985de0675559ce860647d7bf381
Aug 09 20:28:38 dist-test-slave-rnrw krb5kdc[30693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307318, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:28:38.416522 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.398285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:53173 (local address 127.12.78.62:46457)
0809 20:28:38.398785 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:38.398790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:38.399733 (+ 943us) server_negotiation.cc:407] Connection header received
0809 20:28:38.402188 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:38.402194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:38.402279 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:38.402402 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:38.403993 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.404863 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.406385 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.406779 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.406917 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.410532 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:38.410537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:38.410541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:38.410585 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:38.413281 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:38.414128 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:38.414135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:38.414138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:38.414276 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:38.414759 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:38.414765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:38.414768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:38.415047 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:38.415210 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:38.415921 (+ 711us) server_negotiation.cc:299] Negotiation successful
0809 20:28:38.416151 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":249,"threads_started":1}
I20260809 20:28:38.417493 31209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.398792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:46457 (local address 127.12.78.3:53173)
0809 20:28:38.399569 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:38.399607 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:38.401904 (+ 2297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:38.402649 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:38.402658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:38.403208 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:38.403825 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:38.403842 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.405026 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.405031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:38.406239 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:38.406251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.407010 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.407015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:38.407288 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.408036 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:28:38.408057 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:38.410326 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:38.413414 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:38.413421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:38.413432 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:38.414000 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:38.414390 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:38.414395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:38.414399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:38.414652 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:38.415185 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:38.415193 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:38.415530 (+ 337us) client_negotiation.cc:769] Sending connection context
0809 20:28:38.415800 (+ 270us) client_negotiation.cc:240] Negotiation successful
0809 20:28:38.416326 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":315,"threads_started":1}
I20260809 20:28:38.418910 31206 heartbeater.cc:347] Connected to a master server at 127.12.78.62:46457
I20260809 20:28:38.419235 31206 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:38.420018 31206 heartbeater.cc:511] Master 127.12.78.62:46457 requested a full tablet report, sending...
I20260809 20:28:38.421449 30738 ts_manager.cc:194] Registered new tserver with Master: 286dc9f07370473085e1897c8beabcc5 (127.12.78.3:38611)
I20260809 20:28:38.422389 30738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:53173
I20260809 20:28:38.427958 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:28:38.445606 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.432600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.12.78.62:46457)
0809 20:28:38.432817 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:38.432822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:38.433332 (+ 510us) server_negotiation.cc:407] Connection header received
0809 20:28:38.433467 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:38.433471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:38.433556 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:38.433661 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:38.435175 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.436038 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.437451 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.437894 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.438000 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.440526 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:38.440531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:38.440534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:38.440606 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:38.443203 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:38.443829 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:38.443832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:38.443835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:38.443948 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:38.444301 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:38.444305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:38.444308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:38.444714 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:38.444917 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:38.445210 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 20:28:38.445382 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260809 20:28:38.450001 30738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:28:38.457664 30738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:28:38.481453 31219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.469799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:40879 (local address 127.0.0.1:37462)
0809 20:28:38.470714 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:38.470765 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:38.470990 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:38.473442 (+ 2452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:38.473446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:38.473470 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:38.473858 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:38.473870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.475849 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.475857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:38.478244 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:38.478258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.480177 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.480183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:38.480394 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.480464 (+ 70us) client_negotiation.cc:769] Sending connection context
0809 20:28:38.480658 (+ 194us) client_negotiation.cc:240] Negotiation successful
0809 20:28:38.480858 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":114,"thread_start_us":581,"threads_started":1}
I20260809 20:28:38.481546 31220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.469893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:38611 (local address 127.0.0.1:60620)
0809 20:28:38.471472 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:38.471491 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:38.471606 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:38.473067 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:38.473074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:38.473105 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:38.473441 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:38.473452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.475898 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.475901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:38.478717 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:38.478732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.480616 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.480625 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:38.480809 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.480842 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 20:28:38.480936 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 20:28:38.481095 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1413,"mutex_wait_us":102,"thread_start_us":1258,"threads_started":1}
I20260809 20:28:38.481567 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.470621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.12.78.3:38611)
0809 20:28:38.472261 (+ 1640us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:38.472272 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:38.472311 (+ 39us) server_negotiation.cc:407] Connection header received
0809 20:28:38.472431 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:38.472443 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:38.472776 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:38.473661 (+ 885us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:38.473957 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.475697 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.478933 (+ 3236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.480370 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.480504 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.480764 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:38.481009 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 20:28:38.481212 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1426,"thread_start_us":1241,"threads_started":1}
I20260809 20:28:38.482196 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.471557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:40037 (local address 127.0.0.1:35124)
0809 20:28:38.472158 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:38.472176 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:38.472295 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:38.473862 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:38.473866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:38.473893 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:38.474382 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:38.474400 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.477049 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.477054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:38.479291 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:38.479306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.481213 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.481221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:38.481404 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.481442 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 20:28:38.481513 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 20:28:38.481835 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":509,"threads_started":1}
I20260809 20:28:38.482322 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.469965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.12.78.2:40879)
0809 20:28:38.471615 (+ 1650us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:38.471623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:38.471651 (+ 28us) server_negotiation.cc:407] Connection header received
0809 20:28:38.472575 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:38.472585 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:38.473222 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:38.473461 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:38.474104 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.475709 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.478429 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.479981 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.480142 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.480364 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:38.481754 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0809 20:28:38.481957 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":1368,"threads_started":1}
I20260809 20:28:38.483004 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.471881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.12.78.1:40037)
0809 20:28:38.473279 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:38.473287 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:38.473319 (+ 32us) server_negotiation.cc:407] Connection header received
0809 20:28:38.473420 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:38.473427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:38.473648 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:38.473868 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:38.474551 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.476874 (+ 2323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.479487 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.480956 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.482193 (+ 1237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.482450 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:38.482535 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 20:28:38.482696 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1213,"thread_start_us":1034,"threads_started":1}
I20260809 20:28:38.485913 31141 tablet_service.cc:1467] Processing CreateTablet for tablet 7203d2524d5e45d6b307b0254c2f7dbb (DEFAULT_TABLE table=test-table [id=67d4286925b3423ea93b9cd250d757b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:38.486454 30865 tablet_service.cc:1467] Processing CreateTablet for tablet 7203d2524d5e45d6b307b0254c2f7dbb (DEFAULT_TABLE table=test-table [id=67d4286925b3423ea93b9cd250d757b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:38.486454 31004 tablet_service.cc:1467] Processing CreateTablet for tablet 7203d2524d5e45d6b307b0254c2f7dbb (DEFAULT_TABLE table=test-table [id=67d4286925b3423ea93b9cd250d757b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:38.487393 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7203d2524d5e45d6b307b0254c2f7dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:38.487851 31004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7203d2524d5e45d6b307b0254c2f7dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:38.487828 30865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7203d2524d5e45d6b307b0254c2f7dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:38.499874 31225 tablet_bootstrap.cc:492] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5: Bootstrap starting.
I20260809 20:28:38.504302 31225 tablet_bootstrap.cc:654] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:38.505472 31225 log.cc:824] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:38.506065 31227 tablet_bootstrap.cc:492] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b: Bootstrap starting.
I20260809 20:28:38.508625 31226 tablet_bootstrap.cc:492] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b: Bootstrap starting.
I20260809 20:28:38.509353 31225 tablet_bootstrap.cc:492] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5: No bootstrap required, opened a new log
I20260809 20:28:38.509660 31225 ts_tablet_manager.cc:1397] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 20:28:38.510346 31227 tablet_bootstrap.cc:654] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:38.511667 31227 log.cc:824] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:38.512534 31226 tablet_bootstrap.cc:654] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:38.514079 31226 log.cc:824] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:38.514384 31227 tablet_bootstrap.cc:492] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b: No bootstrap required, opened a new log
I20260809 20:28:38.514663 31227 ts_tablet_manager.cc:1397] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 20:28:38.517483 31226 tablet_bootstrap.cc:492] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b: No bootstrap required, opened a new log
I20260809 20:28:38.517810 31226 ts_tablet_manager.cc:1397] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260809 20:28:38.521588 31225 raft_consensus.cc:348] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f956c089ad84f408976d5517488960b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 40879 } } peers { permanent_uuid: "286dc9f07370473085e1897c8beabcc5" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 38611 } } peers { permanent_uuid: "893d554182c645d180d0d187d6a0f62b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40037 } }
I20260809 20:28:38.522241 31225 raft_consensus.cc:374] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:38.522380 31225 raft_consensus.cc:729] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286dc9f07370473085e1897c8beabcc5, State: Initialized, Role: FOLLOWER
I20260809 20:28:38.523113 31227 raft_consensus.cc:348] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f956c089ad84f408976d5517488960b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 40879 } } peers { permanent_uuid: "286dc9f07370473085e1897c8beabcc5" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 38611 } } peers { permanent_uuid: "893d554182c645d180d0d187d6a0f62b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40037 } }
I20260809 20:28:38.523108 31225 consensus_queue.cc:260] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f956c089ad84f408976d5517488960b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 40879 } } peers { permanent_uuid: "286dc9f07370473085e1897c8beabcc5" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 38611 } } peers { permanent_uuid: "893d554182c645d180d0d187d6a0f62b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40037 } }
I20260809 20:28:38.523458 31227 raft_consensus.cc:374] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:38.523561 31227 raft_consensus.cc:729] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f956c089ad84f408976d5517488960b, State: Initialized, Role: FOLLOWER
I20260809 20:28:38.524374 31206 heartbeater.cc:503] Master 127.12.78.62:46457 was elected leader, sending a full tablet report...
I20260809 20:28:38.525151 31225 ts_tablet_manager.cc:1428] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260809 20:28:38.525601 31227 consensus_queue.cc:260] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f956c089ad84f408976d5517488960b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 40879 } } peers { permanent_uuid: "286dc9f07370473085e1897c8beabcc5" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 38611 } } peers { permanent_uuid: "893d554182c645d180d0d187d6a0f62b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40037 } }
I20260809 20:28:38.526773 31070 heartbeater.cc:503] Master 127.12.78.62:46457 was elected leader, sending a full tablet report...
I20260809 20:28:38.528934 31226 raft_consensus.cc:348] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f956c089ad84f408976d5517488960b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 40879 } } peers { permanent_uuid: "286dc9f07370473085e1897c8beabcc5" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 38611 } } peers { permanent_uuid: "893d554182c645d180d0d187d6a0f62b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40037 } }
I20260809 20:28:38.529564 31226 raft_consensus.cc:374] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:38.529712 31226 raft_consensus.cc:729] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893d554182c645d180d0d187d6a0f62b, State: Initialized, Role: FOLLOWER
I20260809 20:28:38.529755 31227 ts_tablet_manager.cc:1428] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b: Time spent starting tablet: real 0.015s user 0.010s sys 0.001s
I20260809 20:28:38.530496 31226 consensus_queue.cc:260] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f956c089ad84f408976d5517488960b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 40879 } } peers { permanent_uuid: "286dc9f07370473085e1897c8beabcc5" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 38611 } } peers { permanent_uuid: "893d554182c645d180d0d187d6a0f62b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40037 } }
I20260809 20:28:38.532287 31226 ts_tablet_manager.cc:1428] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260809 20:28:38.584134 30932 tablet.cc:2369] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:28:38.612886 31071 tablet.cc:2369] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:28:38.647403 31207 tablet.cc:2369] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:28:38.656783 31231 raft_consensus.cc:482] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:28:38.657049 31231 raft_consensus.cc:504] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f956c089ad84f408976d5517488960b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 40879 } } peers { permanent_uuid: "286dc9f07370473085e1897c8beabcc5" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 38611 } } peers { permanent_uuid: "893d554182c645d180d0d187d6a0f62b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40037 } }
I20260809 20:28:38.658627 31231 leader_election.cc:288] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f956c089ad84f408976d5517488960b (127.12.78.2:40879), 893d554182c645d180d0d187d6a0f62b (127.12.78.1:40037)
I20260809 20:28:38.665623 31209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.658894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:40879 (local address 127.12.78.3:57459)
0809 20:28:38.659087 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:38.659110 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:38.659244 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:38.659703 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:38.659708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:38.659738 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:38.659948 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:38.659956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.661265 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.661273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:38.663182 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:38.663203 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.664843 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.664849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:38.665050 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.665101 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 20:28:38.665205 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 20:28:38.665352 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":2}
I20260809 20:28:38.665699 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.659223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:57459 (local address 127.12.78.2:40879)
0809 20:28:38.659420 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:38.659424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:38.659442 (+ 18us) server_negotiation.cc:407] Connection header received
0809 20:28:38.659502 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:38.659506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:38.659563 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:38.659678 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:38.660114 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.661132 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.663349 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.664657 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.664787 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.664882 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:38.665319 (+ 437us) server_negotiation.cc:299] Negotiation successful
0809 20:28:38.665451 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 20:28:38.665956 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.659126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:35751 (local address 127.12.78.1:40037)
0809 20:28:38.659359 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:38.659366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:38.660187 (+ 821us) server_negotiation.cc:407] Connection header received
0809 20:28:38.660295 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:38.660300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:38.660376 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:38.660464 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:38.661019 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.661937 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.664009 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.664967 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.665105 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.665227 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:38.665578 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 20:28:38.665714 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 20:28:38.666628 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.658894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:40037 (local address 127.12.78.3:35751)
0809 20:28:38.660090 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:38.660104 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:38.660200 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:38.660491 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:38.660494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:38.660511 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:38.660862 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:38.660870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.662103 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.662109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:38.663860 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:38.663872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.665188 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.665194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:38.665372 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.665404 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 20:28:38.666141 (+ 737us) client_negotiation.cc:240] Negotiation successful
0809 20:28:38.666271 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":484,"thread_start_us":391,"threads_started":1}
I20260809 20:28:38.667039 31024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7203d2524d5e45d6b307b0254c2f7dbb" candidate_uuid: "286dc9f07370473085e1897c8beabcc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f956c089ad84f408976d5517488960b" is_pre_election: true
I20260809 20:28:38.667680 31024 raft_consensus.cc:2454] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 286dc9f07370473085e1897c8beabcc5 in term 0.
I20260809 20:28:38.667594 30885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7203d2524d5e45d6b307b0254c2f7dbb" candidate_uuid: "286dc9f07370473085e1897c8beabcc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893d554182c645d180d0d187d6a0f62b" is_pre_election: true
I20260809 20:28:38.668140 30885 raft_consensus.cc:2454] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 286dc9f07370473085e1897c8beabcc5 in term 0.
I20260809 20:28:38.668448 31095 leader_election.cc:302] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286dc9f07370473085e1897c8beabcc5, 6f956c089ad84f408976d5517488960b; no voters:
I20260809 20:28:38.668874 31231 raft_consensus.cc:2781] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:28:38.669020 31231 raft_consensus.cc:482] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:28:38.669129 31231 raft_consensus.cc:3037] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:38.671293 31231 raft_consensus.cc:504] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f956c089ad84f408976d5517488960b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 40879 } } peers { permanent_uuid: "286dc9f07370473085e1897c8beabcc5" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 38611 } } peers { permanent_uuid: "893d554182c645d180d0d187d6a0f62b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40037 } }
I20260809 20:28:38.671965 31231 leader_election.cc:288] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [CANDIDATE]: Term 1 election: Requested vote from peers 6f956c089ad84f408976d5517488960b (127.12.78.2:40879), 893d554182c645d180d0d187d6a0f62b (127.12.78.1:40037)
I20260809 20:28:38.672396 31024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7203d2524d5e45d6b307b0254c2f7dbb" candidate_uuid: "286dc9f07370473085e1897c8beabcc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f956c089ad84f408976d5517488960b"
I20260809 20:28:38.672515 30885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7203d2524d5e45d6b307b0254c2f7dbb" candidate_uuid: "286dc9f07370473085e1897c8beabcc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893d554182c645d180d0d187d6a0f62b"
I20260809 20:28:38.672622 31024 raft_consensus.cc:3037] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:38.672756 30885 raft_consensus.cc:3037] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:38.675083 31024 raft_consensus.cc:2454] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286dc9f07370473085e1897c8beabcc5 in term 1.
I20260809 20:28:38.675488 31095 leader_election.cc:302] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286dc9f07370473085e1897c8beabcc5, 6f956c089ad84f408976d5517488960b; no voters:
I20260809 20:28:38.675800 31231 raft_consensus.cc:2781] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:28:38.675976 30885 raft_consensus.cc:2454] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286dc9f07370473085e1897c8beabcc5 in term 1.
I20260809 20:28:38.679876 31231 raft_consensus.cc:686] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [term 1 LEADER]: Becoming Leader. State: Replica: 286dc9f07370473085e1897c8beabcc5, State: Running, Role: LEADER
I20260809 20:28:38.680371 31231 consensus_queue.cc:237] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f956c089ad84f408976d5517488960b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 40879 } } peers { permanent_uuid: "286dc9f07370473085e1897c8beabcc5" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 38611 } } peers { permanent_uuid: "893d554182c645d180d0d187d6a0f62b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40037 } }
I20260809 20:28:38.685220 30737 catalog_manager.cc:5495] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 286dc9f07370473085e1897c8beabcc5 (127.12.78.3). New cstate: current_term: 1 leader_uuid: "286dc9f07370473085e1897c8beabcc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f956c089ad84f408976d5517488960b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 40879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "286dc9f07370473085e1897c8beabcc5" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 38611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "893d554182c645d180d0d187d6a0f62b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40037 } health_report { overall_health: UNKNOWN } } }
I20260809 20:28:38.720371 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.714852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.12.78.3:38611)
0809 20:28:38.715067 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:38.715074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:38.715092 (+ 18us) server_negotiation.cc:407] Connection header received
0809 20:28:38.715183 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:38.715187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:38.715259 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:38.715411 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:28:38.715805 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.716503 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.718190 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.718681 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.718771 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.719305 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:28:38.719822 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:28:38.719939 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:38.720023 (+ 84us) server_negotiation.cc:299] Negotiation successful
0809 20:28:38.720089 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 20:28:38.728430 31024 raft_consensus.cc:1261] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b [term 1 FOLLOWER]: Refusing update from remote peer 286dc9f07370473085e1897c8beabcc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:28:38.728428 30885 raft_consensus.cc:1261] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b [term 1 FOLLOWER]: Refusing update from remote peer 286dc9f07370473085e1897c8beabcc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:28:38.729415 31236 consensus_queue.cc:1035] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f956c089ad84f408976d5517488960b" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:28:38.729821 31231 consensus_queue.cc:1035] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "893d554182c645d180d0d187d6a0f62b" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 40037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:28:38.740662 31239 mvcc.cc:205] Tried to move back new op lower bound from 7316714777501270016 to 7316714777318486016. Current Snapshot: MvccSnapshot[applied={T|T < 7316714777501270016}]
I20260809 20:28:38.753173 30738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:28:38.753485 30738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:28:38.757022 30738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bfceda790084b41aab898e09d209aa5: Sending DeleteTablet for 3 replicas of tablet 7203d2524d5e45d6b307b0254c2f7dbb
I20260809 20:28:38.758119 31004 tablet_service.cc:1514] Processing DeleteTablet for tablet 7203d2524d5e45d6b307b0254c2f7dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:38 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:37462
I20260809 20:28:38.758433 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet 7203d2524d5e45d6b307b0254c2f7dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:38 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:60620
I20260809 20:28:38.758649 30865 tablet_service.cc:1514] Processing DeleteTablet for tablet 7203d2524d5e45d6b307b0254c2f7dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:38 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:35124
I20260809 20:28:38.762770 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 30802
I20260809 20:28:38.768877 31251 tablet_replica.cc:330] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b: stopping tablet replica
I20260809 20:28:38.769089 31249 tablet_replica.cc:330] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b: stopping tablet replica
I20260809 20:28:38.769352 31251 raft_consensus.cc:2229] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:28:38.769461 31249 raft_consensus.cc:2229] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:28:38.769917 31249 raft_consensus.cc:2258] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:38.769917 31251 raft_consensus.cc:2258] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:38.771844 31250 tablet_replica.cc:330] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5: stopping tablet replica
I20260809 20:28:38.772270 31251 ts_tablet_manager.cc:1905] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:38.772236 31250 raft_consensus.cc:2229] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:28:38.772943 31250 raft_consensus.cc:2258] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:38.774776 31249 ts_tablet_manager.cc:1905] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:38.778815 31251 ts_tablet_manager.cc:1918] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:38.779006 31251 log.cc:1196] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/7203d2524d5e45d6b307b0254c2f7dbb
I20260809 20:28:38.779606 31251 ts_tablet_manager.cc:1939] T 7203d2524d5e45d6b307b0254c2f7dbb P 893d554182c645d180d0d187d6a0f62b: Deleting consensus metadata
I20260809 20:28:38.781548 30722 catalog_manager.cc:4841] TS 893d554182c645d180d0d187d6a0f62b (127.12.78.1:40037): tablet 7203d2524d5e45d6b307b0254c2f7dbb (table test-table [id=67d4286925b3423ea93b9cd250d757b1]) successfully deleted
I20260809 20:28:38.782743 31250 ts_tablet_manager.cc:1905] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 20:28:38 dist-test-slave-rnrw krb5kdc[30693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:28:38.786165 31249 ts_tablet_manager.cc:1918] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:38.786414 31249 log.cc:1196] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/7203d2524d5e45d6b307b0254c2f7dbb
I20260809 20:28:38.787381 31249 ts_tablet_manager.cc:1939] T 7203d2524d5e45d6b307b0254c2f7dbb P 6f956c089ad84f408976d5517488960b: Deleting consensus metadata
I20260809 20:28:38.789674 30725 catalog_manager.cc:4841] TS 6f956c089ad84f408976d5517488960b (127.12.78.2:40879): tablet 7203d2524d5e45d6b307b0254c2f7dbb (table test-table [id=67d4286925b3423ea93b9cd250d757b1]) successfully deleted
I20260809 20:28:38.790231 31250 ts_tablet_manager.cc:1918] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:38.790402 31250 log.cc:1196] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/7203d2524d5e45d6b307b0254c2f7dbb
I20260809 20:28:38.790930 31250 ts_tablet_manager.cc:1939] T 7203d2524d5e45d6b307b0254c2f7dbb P 286dc9f07370473085e1897c8beabcc5: Deleting consensus metadata
I20260809 20:28:38.792624 30722 catalog_manager.cc:4841] TS 286dc9f07370473085e1897c8beabcc5 (127.12.78.3:38611): tablet 7203d2524d5e45d6b307b0254c2f7dbb (table test-table [id=67d4286925b3423ea93b9cd250d757b1]) successfully deleted
I20260809 20:28:38.793972 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.764203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.12.78.1:40037)
0809 20:28:38.764389 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:38.764395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:38.771253 (+ 6858us) server_negotiation.cc:407] Connection header received
0809 20:28:38.771388 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:38.771395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:38.771529 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:38.771650 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:38.775007 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.776197 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.778523 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.779385 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.779496 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.786116 (+ 6620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:38.786132 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:38.786152 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:38.786207 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:38.790349 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:38.791287 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:38.791296 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:38.791300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:38.791472 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:38.791952 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:38.791957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:38.791960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:38.792425 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:38.793112 (+ 687us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:38.793349 (+ 237us) server_negotiation.cc:299] Negotiation successful
0809 20:28:38.793573 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 20:28:38.794620 30926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:38.925866 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 30802
I20260809 20:28:38.945267 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 30938
Aug 09 20:28:38 dist-test-slave-rnrw krb5kdc[30693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:28:38.961997 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:38.946046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.12.78.2:40879)
0809 20:28:38.946235 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:38.946241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:38.946313 (+ 72us) server_negotiation.cc:407] Connection header received
0809 20:28:38.946481 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:38.946487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:38.946580 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:38.946703 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:38.948066 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.949238 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.950501 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:38.951116 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:38.951243 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:38.955463 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:38.955481 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:38.955501 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:38.955546 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:38.959216 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:38.959864 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:38.959874 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:38.959878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:38.960032 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:38.960361 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:38.960366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:38.960370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:38.960877 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:38.961165 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:38.961366 (+ 201us) server_negotiation.cc:299] Negotiation successful
0809 20:28:38.961599 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 20:28:38.962710 31065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:39.052248 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 30938
I20260809 20:28:39.071529 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 31076
Aug 09 20:28:39 dist-test-slave-rnrw krb5kdc[30693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:28:39.092001 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:39.072430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.12.78.3:38611)
0809 20:28:39.072685 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:39.072691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:39.072714 (+ 23us) server_negotiation.cc:407] Connection header received
0809 20:28:39.072835 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:39.072841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:39.072927 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:39.073061 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:39.074369 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:39.075676 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:39.076973 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:39.077659 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:39.077787 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:39.082828 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:39.082847 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:39.082869 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:39.082918 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:39.087277 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:39.088265 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:39.088279 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:39.088283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:39.088489 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:39.089020 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:39.089025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:39.089028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:39.089519 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:39.089882 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:39.091299 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0809 20:28:39.091565 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 20:28:39.093273 31201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:39.217185 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 31076
I20260809 20:28:39.239336 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 30706
I20260809 20:28:39.240492 30769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:39.353219 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 30706
2026-08-09T20:28:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4695 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T20:28:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:28:39Z Disabled control of system clock
I20260809 20:28:39.448962 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:33407
--webserver_interface=127.12.78.62
--webserver_port=0
--builtin_ntp_servers=127.12.78.20:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:33407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 20:28:39.795265 31261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:39.795558 31261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:39.795641 31261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:39.802839 31261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:28:39.802951 31261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:39.802991 31261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:39.803050 31261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:28:39.803095 31261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:28:39.812197 31261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:33407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:33407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:39.813515 31261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:39.815397 31261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:39.827072 31269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:39.827126 31267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:39.827286 31266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:39.828233 31261 server_base.cc:1034] running on GCE node
I20260809 20:28:39.829012 31261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:39.830307 31261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:39.831537 31261 hybrid_clock.cc:648] HybridClock initialized: now 1786307319831494 us; error 52 us; skew 500 ppm
I20260809 20:28:39.831885 31261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:28:39.838040 31261 webserver.cc:460] Webserver started at http://127.12.78.62:37935/ using document root <none> and password file <none>
I20260809 20:28:39.838752 31261 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:39.838891 31261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:39.839221 31261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:39.842243 31261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "4862e8d746b64503a15705979aa34db2"
format_stamp: "Formatted at 2026-08-09 20:28:39 on dist-test-slave-rnrw"
server_key: "e411f7845a06ece758cd3f22d615fc29"
server_key_iv: "ab9c4922ed8f1790e638e09f4abb1cea"
server_key_version: "encryptionkey@0"
I20260809 20:28:39.843048 31261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "4862e8d746b64503a15705979aa34db2"
format_stamp: "Formatted at 2026-08-09 20:28:39 on dist-test-slave-rnrw"
server_key: "e411f7845a06ece758cd3f22d615fc29"
server_key_iv: "ab9c4922ed8f1790e638e09f4abb1cea"
server_key_version: "encryptionkey@0"
I20260809 20:28:39.848129 31261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 20:28:39.851663 31276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:39.853048 31261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:28:39.853256 31261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "4862e8d746b64503a15705979aa34db2"
format_stamp: "Formatted at 2026-08-09 20:28:39 on dist-test-slave-rnrw"
server_key: "e411f7845a06ece758cd3f22d615fc29"
server_key_iv: "ab9c4922ed8f1790e638e09f4abb1cea"
server_key_version: "encryptionkey@0"
I20260809 20:28:39.853473 31261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:39.872042 31261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:39.872960 31261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:39.873270 31261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:39.892310 31261 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:33407
I20260809 20:28:39.892336 31328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:33407 every 8 connection(s)
I20260809 20:28:39.893678 31261 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:28:39.896485 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 31261
I20260809 20:28:39.896734 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:28:39.897197 12600 external_mini_cluster.cc:1428] Setting key ce3bddae702cc6cd72e71508fc3fd603
I20260809 20:28:39.897650 31329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:39.910006 31332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:39.898888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.12.78.62:33407)
0809 20:28:39.899750 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:39.899768 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:39.899809 (+ 41us) server_negotiation.cc:407] Connection header received
0809 20:28:39.901436 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:39.901458 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:39.901897 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:39.902305 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:39.902874 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:39.904189 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:39.905615 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:39.906270 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:39.906463 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:39.907685 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:39.907692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:39.907704 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:39.908266 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:39.908343 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:39.908372 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:39.908522 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:39.908777 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 20:28:39.909087 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":291,"threads_started":1}
I20260809 20:28:39.910725 31329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2: Bootstrap starting.
I20260809 20:28:39.913941 31329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:39.914963 31329 log.cc:824] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:39.917963 31329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2: No bootstrap required, opened a new log
I20260809 20:28:39.925575 31329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4862e8d746b64503a15705979aa34db2" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33407 } }
I20260809 20:28:39.925969 31329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:39.926075 31329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4862e8d746b64503a15705979aa34db2, State: Initialized, Role: FOLLOWER
I20260809 20:28:39.926666 31329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4862e8d746b64503a15705979aa34db2" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33407 } }
I20260809 20:28:39.926904 31329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:28:39.927054 31329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:28:39.927228 31329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:39.929759 31329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4862e8d746b64503a15705979aa34db2" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33407 } }
I20260809 20:28:39.930254 31329 leader_election.cc:302] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4862e8d746b64503a15705979aa34db2; no voters:
I20260809 20:28:39.930871 31329 leader_election.cc:288] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:28:39.931003 31334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:28:39.931754 31334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [term 1 LEADER]: Becoming Leader. State: Replica: 4862e8d746b64503a15705979aa34db2, State: Running, Role: LEADER
I20260809 20:28:39.932301 31334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4862e8d746b64503a15705979aa34db2" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33407 } }
I20260809 20:28:39.933012 31329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:28:39.935137 31335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4862e8d746b64503a15705979aa34db2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4862e8d746b64503a15705979aa34db2" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33407 } } }
I20260809 20:28:39.935529 31335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:39.935954 31336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4862e8d746b64503a15705979aa34db2. Latest consensus state: current_term: 1 leader_uuid: "4862e8d746b64503a15705979aa34db2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4862e8d746b64503a15705979aa34db2" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33407 } } }
I20260809 20:28:39.936275 31336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:39.941709 31340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:28:39.947533 31340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:28:39.956229 31340 catalog_manager.cc:1347] Generated new cluster ID: 70e620a7a3f64a3ca4c97c3029609992
I20260809 20:28:39.956353 31340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:28:39.966504 31340 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:28:39.967980 31340 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:28:39.978478 31340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2: Generated new TSK 0
I20260809 20:28:39.979496 31340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:28:39.994043 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33407
--builtin_ntp_servers=127.12.78.20:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 20:28:40.339653 31353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:40.339926 31353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:40.340070 31353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:40.347399 31353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:40.347584 31353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:40.347749 31353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:28:40.357487 31353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:40.358860 31353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:40.360752 31353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:40.374135 31358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:40.374243 31359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:40.374344 31361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:40.375130 31353 server_base.cc:1034] running on GCE node
I20260809 20:28:40.375708 31353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:40.376672 31353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:40.377911 31353 hybrid_clock.cc:648] HybridClock initialized: now 1786307320377870 us; error 34 us; skew 500 ppm
I20260809 20:28:40.378247 31353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:28:40.380602 31353 webserver.cc:460] Webserver started at http://127.12.78.1:41085/ using document root <none> and password file <none>
I20260809 20:28:40.381289 31353 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:40.381424 31353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:40.381754 31353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:40.384657 31353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "6a6cf69d141d4ab9a9652df25991184f"
format_stamp: "Formatted at 2026-08-09 20:28:40 on dist-test-slave-rnrw"
server_key: "e6864532b05ed97e492f920f21dcbc12"
server_key_iv: "43457f9cb8e12d2cb0d9da5824b65259"
server_key_version: "encryptionkey@0"
I20260809 20:28:40.385460 31353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "6a6cf69d141d4ab9a9652df25991184f"
format_stamp: "Formatted at 2026-08-09 20:28:40 on dist-test-slave-rnrw"
server_key: "e6864532b05ed97e492f920f21dcbc12"
server_key_iv: "43457f9cb8e12d2cb0d9da5824b65259"
server_key_version: "encryptionkey@0"
I20260809 20:28:40.390590 31353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260809 20:28:40.394171 31367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:40.395481 31353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 20:28:40.395653 31353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "6a6cf69d141d4ab9a9652df25991184f"
format_stamp: "Formatted at 2026-08-09 20:28:40 on dist-test-slave-rnrw"
server_key: "e6864532b05ed97e492f920f21dcbc12"
server_key_iv: "43457f9cb8e12d2cb0d9da5824b65259"
server_key_version: "encryptionkey@0"
I20260809 20:28:40.395864 31353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:40.409883 31353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:40.410672 31353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:40.410950 31353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:40.411878 31353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:40.413364 31353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:40.413493 31353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:40.413615 31353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:40.413699 31353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:40.448701 31353 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:36769
I20260809 20:28:40.448761 31484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:36769 every 8 connection(s)
I20260809 20:28:40.450311 31353 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:28:40.455449 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 31353
I20260809 20:28:40.455672 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:28:40.456272 12600 external_mini_cluster.cc:1428] Setting key ccac6f189a74f3546305b8250bf69638
I20260809 20:28:40.459383 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33407
--builtin_ntp_servers=127.12.78.20:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 20:28:40.465369 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:40.453105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:33769 (local address 127.12.78.62:33407)
0809 20:28:40.453602 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:40.453609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:40.454911 (+ 1302us) server_negotiation.cc:407] Connection header received
0809 20:28:40.457415 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:40.457421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:40.457514 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:40.457636 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:40.458856 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:40.459709 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:40.462352 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:40.462790 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:40.462892 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:40.464591 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:40.464602 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:40.464611 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:40.464630 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:40.464671 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:40.464685 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:40.464762 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:40.465017 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 20:28:40.465102 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":224,"threads_started":1}
I20260809 20:28:40.466871 31488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:40.453811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:33407 (local address 127.12.78.1:33769)
0809 20:28:40.454741 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:40.454784 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:40.457136 (+ 2352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:40.457887 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:40.457896 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:40.457987 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:40.458666 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:40.458691 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:40.459871 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:40.459879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:40.462182 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:40.462200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:40.463018 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:40.463027 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:40.463365 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:40.464175 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:28:40.464182 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:40.464237 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:28:40.464341 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:40.464780 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:40.464795 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:40.464805 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 20:28:40.464996 (+ 191us) client_negotiation.cc:240] Negotiation successful
0809 20:28:40.465436 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":302,"threads_started":1}
I20260809 20:28:40.468483 31485 heartbeater.cc:347] Connected to a master server at 127.12.78.62:33407
I20260809 20:28:40.468830 31485 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:40.469766 31485 heartbeater.cc:511] Master 127.12.78.62:33407 requested a full tablet report, sending...
I20260809 20:28:40.471916 31293 ts_manager.cc:194] Registered new tserver with Master: 6a6cf69d141d4ab9a9652df25991184f (127.12.78.1:36769)
I20260809 20:28:40.473500 31293 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.78.1:33769
W20260809 20:28:40.803980 31489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:40.804260 31489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:40.804407 31489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:40.811596 31489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:40.811694 31489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:40.811866 31489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:28:40.821058 31489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:40.822336 31489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:40.824096 31489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:40.837208 31494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:40.837352 31495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:40.837183 31497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:40.841964 31489 server_base.cc:1034] running on GCE node
I20260809 20:28:40.842558 31489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:40.843523 31489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:40.844720 31489 hybrid_clock.cc:648] HybridClock initialized: now 1786307320844670 us; error 41 us; skew 500 ppm
I20260809 20:28:40.845036 31489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:28:40.847267 31489 webserver.cc:460] Webserver started at http://127.12.78.2:44105/ using document root <none> and password file <none>
I20260809 20:28:40.847957 31489 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:40.848070 31489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:40.848381 31489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:40.851293 31489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "c42d90936b174bfeadbe526e0154a9b7"
format_stamp: "Formatted at 2026-08-09 20:28:40 on dist-test-slave-rnrw"
server_key: "5e2ebb39ee6fd3c41351a9f36a4cfd8a"
server_key_iv: "44d3423d22251ff79292009082decf00"
server_key_version: "encryptionkey@0"
I20260809 20:28:40.852072 31489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "c42d90936b174bfeadbe526e0154a9b7"
format_stamp: "Formatted at 2026-08-09 20:28:40 on dist-test-slave-rnrw"
server_key: "5e2ebb39ee6fd3c41351a9f36a4cfd8a"
server_key_iv: "44d3423d22251ff79292009082decf00"
server_key_version: "encryptionkey@0"
I20260809 20:28:40.857280 31489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 20:28:40.860920 31503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:40.862323 31489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 20:28:40.862500 31489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "c42d90936b174bfeadbe526e0154a9b7"
format_stamp: "Formatted at 2026-08-09 20:28:40 on dist-test-slave-rnrw"
server_key: "5e2ebb39ee6fd3c41351a9f36a4cfd8a"
server_key_iv: "44d3423d22251ff79292009082decf00"
server_key_version: "encryptionkey@0"
I20260809 20:28:40.862728 31489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:40.892035 31489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:40.892944 31489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:40.893225 31489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:40.894467 31489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:40.895960 31489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:40.896097 31489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:40.896221 31489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:40.896306 31489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:40.930349 31489 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:43075
I20260809 20:28:40.930433 31619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:43075 every 8 connection(s)
I20260809 20:28:40.931653 31489 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:28:40.941309 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 31489
I20260809 20:28:40.941506 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:28:40.942070 12600 external_mini_cluster.cc:1428] Setting key 74049113c445f9ee397b83d94066d7a0
I20260809 20:28:40.945331 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:40.934300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:44323 (local address 127.12.78.62:33407)
0809 20:28:40.934504 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:40.934508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:40.935922 (+ 1414us) server_negotiation.cc:407] Connection header received
0809 20:28:40.938003 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:40.938010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:40.938123 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:40.938240 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:40.939357 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:40.940087 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:40.942285 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:40.942718 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:40.942825 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:40.944417 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:40.944423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:40.944427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:40.944447 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:40.944489 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:40.944504 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:40.944658 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:40.944871 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 20:28:40.944967 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 20:28:40.945556 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33407
--builtin_ntp_servers=127.12.78.20:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 20:28:40.946339 31622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:40.934828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:33407 (local address 127.12.78.2:44323)
0809 20:28:40.935749 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:40.935789 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:40.937710 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:40.938480 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:40.938489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:40.938577 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:40.939180 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:40.939202 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:40.940265 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:40.940269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:40.942129 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:40.942146 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:40.942953 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:40.942962 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:40.943301 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:40.944031 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:28:40.944034 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:40.944097 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:28:40.944214 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:40.944658 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:40.944667 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:40.944673 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 20:28:40.944830 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 20:28:40.945159 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":288,"threads_started":1}
I20260809 20:28:40.947778 31620 heartbeater.cc:347] Connected to a master server at 127.12.78.62:33407
I20260809 20:28:40.948068 31620 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:40.948912 31620 heartbeater.cc:511] Master 127.12.78.62:33407 requested a full tablet report, sending...
I20260809 20:28:40.950258 31293 ts_manager.cc:194] Registered new tserver with Master: c42d90936b174bfeadbe526e0154a9b7 (127.12.78.2:43075)
I20260809 20:28:40.951037 31293 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.78.2:44323
W20260809 20:28:41.283068 31623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:41.283880 31623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:41.284044 31623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:41.292884 31623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:41.293004 31623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:41.293135 31623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:28:41.303741 31623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:33407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:41.305173 31623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:41.307052 31623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:41.321322 31631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:41.321352 31629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:41.321544 31628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:41.321940 31623 server_base.cc:1034] running on GCE node
I20260809 20:28:41.322712 31623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:41.323585 31623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:41.324779 31623 hybrid_clock.cc:648] HybridClock initialized: now 1786307321324740 us; error 31 us; skew 500 ppm
I20260809 20:28:41.325089 31623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:28:41.327255 31623 webserver.cc:460] Webserver started at http://127.12.78.3:44435/ using document root <none> and password file <none>
I20260809 20:28:41.327951 31623 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:41.328047 31623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:41.328397 31623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:41.331300 31623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "dcd869f501a44f578098dd72efcd0614"
format_stamp: "Formatted at 2026-08-09 20:28:41 on dist-test-slave-rnrw"
server_key: "96d6d52a12914ea2a5c0dc0c7481e96f"
server_key_iv: "fa42cd618ead4df093a849deb8a48af3"
server_key_version: "encryptionkey@0"
I20260809 20:28:41.332075 31623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "dcd869f501a44f578098dd72efcd0614"
format_stamp: "Formatted at 2026-08-09 20:28:41 on dist-test-slave-rnrw"
server_key: "96d6d52a12914ea2a5c0dc0c7481e96f"
server_key_iv: "fa42cd618ead4df093a849deb8a48af3"
server_key_version: "encryptionkey@0"
I20260809 20:28:41.337533 31623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 20:28:41.341095 31637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:41.342394 31623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 20:28:41.342549 31623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "dcd869f501a44f578098dd72efcd0614"
format_stamp: "Formatted at 2026-08-09 20:28:41 on dist-test-slave-rnrw"
server_key: "96d6d52a12914ea2a5c0dc0c7481e96f"
server_key_iv: "fa42cd618ead4df093a849deb8a48af3"
server_key_version: "encryptionkey@0"
I20260809 20:28:41.342790 31623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:41.363620 31623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:41.364450 31623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:41.364769 31623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:41.365834 31623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:41.367465 31623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:41.367592 31623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:41.367713 31623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:41.367805 31623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:41.401628 31623 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:33363
I20260809 20:28:41.401716 31754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:33363 every 8 connection(s)
I20260809 20:28:41.402956 31623 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:28:41.404220 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 31623
I20260809 20:28:41.404402 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:28:41.405056 12600 external_mini_cluster.cc:1428] Setting key bcfcff0038bb64888feaf6265eabc345
I20260809 20:28:41.415337 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:41.405877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:42579 (local address 127.12.78.62:33407)
0809 20:28:41.406091 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:41.406096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:41.407680 (+ 1584us) server_negotiation.cc:407] Connection header received
0809 20:28:41.409509 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:41.409513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:41.409576 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:41.409659 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:41.410704 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.411426 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.413019 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.413424 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.413518 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:41.414721 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:41.414725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:41.414728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:41.414744 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:41.414769 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:41.414779 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:41.414846 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:41.415022 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 20:28:41.415092 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 20:28:41.416311 31757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:41.406454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:33407 (local address 127.12.78.3:42579)
0809 20:28:41.407507 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:41.407568 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:41.409263 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:41.409842 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:41.409849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:41.409936 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:41.410471 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:41.410487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.411579 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.411583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:41.412889 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:41.412901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.413604 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.413609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:41.413805 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:41.414455 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:28:41.414458 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:41.414494 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:28:41.414577 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:41.414856 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:41.414863 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:41.414868 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 20:28:41.414999 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 20:28:41.415259 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":347,"threads_started":1}
I20260809 20:28:41.417657 31755 heartbeater.cc:347] Connected to a master server at 127.12.78.62:33407
I20260809 20:28:41.417963 31755 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:41.418884 31755 heartbeater.cc:511] Master 127.12.78.62:33407 requested a full tablet report, sending...
I20260809 20:28:41.420406 31293 ts_manager.cc:194] Registered new tserver with Master: dcd869f501a44f578098dd72efcd0614 (127.12.78.3:33363)
I20260809 20:28:41.421253 31293 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.78.3:42579
I20260809 20:28:41.432413 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:28:41.442472 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:41.436704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50990 (local address 127.12.78.62:33407)
0809 20:28:41.436863 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:41.436867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:41.437338 (+ 471us) server_negotiation.cc:407] Connection header received
0809 20:28:41.437493 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:41.437497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:41.437570 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:41.437671 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:41.438119 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.438808 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.440252 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.440731 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.440828 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:41.441859 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:41.441863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:41.441867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:41.441884 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:41.441915 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:41.441927 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:41.441994 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:41.442147 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 20:28:41.442221 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 20:28:41.445986 31293 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:28:41.452889 31293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:28:41.476393 31485 heartbeater.cc:503] Master 127.12.78.62:33407 was elected leader, sending a full tablet report...
I20260809 20:28:41.478435 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:41.465130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:33363 (local address 127.0.0.1:50904)
0809 20:28:41.466252 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:41.466303 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:41.466536 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:41.468058 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:41.468066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:41.468106 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:41.468655 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:41.468675 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.471613 (+ 2938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.471622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:41.474785 (+ 3163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:41.474813 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.477100 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.477114 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:41.477453 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:41.477540 (+ 87us) client_negotiation.cc:769] Sending connection context
0809 20:28:41.477674 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 20:28:41.477949 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":1079,"threads_started":1}
I20260809 20:28:41.478921 31770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:41.465483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.12.78.3:33363)
0809 20:28:41.467299 (+ 1816us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:41.467308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:41.467356 (+ 48us) server_negotiation.cc:407] Connection header received
0809 20:28:41.467479 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:41.467488 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:41.467761 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:41.468910 (+ 1149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:41.469272 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.471417 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.475093 (+ 3676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.476862 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.477960 (+ 1098us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:41.478278 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:41.478352 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 20:28:41.478519 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1631,"thread_start_us":1409,"threads_started":1}
I20260809 20:28:41.483868 31687 tablet_service.cc:1467] Processing CreateTablet for tablet 04e3cdda403548a48f09355960c8daac (DEFAULT_TABLE table=test-table [id=deba7c24ed334875a139a4783a6b352b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:41.485419 31687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e3cdda403548a48f09355960c8daac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:41.486918 31772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:41.465315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:36769 (local address 127.0.0.1:57864)
0809 20:28:41.477592 (+ 12277us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:41.477615 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:41.479293 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:41.480310 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:41.480314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:41.480341 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:41.480865 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:41.480876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.482150 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.482155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:41.484687 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:41.484698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.486080 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.486087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:41.486260 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:41.486302 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 20:28:41.486395 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 20:28:41.486542 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12075,"mutex_wait_us":156,"thread_start_us":12085,"threads_started":1}
I20260809 20:28:41.488006 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:41.465915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.12.78.1:36769)
0809 20:28:41.477041 (+ 11126us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:41.477054 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:41.477730 (+ 676us) server_negotiation.cc:407] Connection header received
0809 20:28:41.479506 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:41.479514 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:41.479793 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:41.480023 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:41.481023 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.482011 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.484884 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.485888 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.486044 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:41.486323 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:41.487285 (+ 962us) server_negotiation.cc:299] Negotiation successful
0809 20:28:41.487487 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10833,"thread_start_us":11125,"threads_started":1}
I20260809 20:28:41.487490 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:41.467772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:43075 (local address 127.0.0.1:60278)
0809 20:28:41.477637 (+ 9865us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:41.477655 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:41.477773 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:41.480062 (+ 2289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:41.480067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:41.480096 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:41.480452 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:41.480462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.482342 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.482347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:41.484611 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:41.484628 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.486468 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.486474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:41.486872 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:41.486922 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 20:28:41.487006 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 20:28:41.487132 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9753,"mutex_wait_us":28,"thread_start_us":9971,"threads_started":1}
I20260809 20:28:41.488243 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:41.467983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60278 (local address 127.12.78.2:43075)
0809 20:28:41.479400 (+ 11417us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:41.479409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:41.479437 (+ 28us) server_negotiation.cc:407] Connection header received
0809 20:28:41.479553 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:41.479565 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:41.479790 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:41.480062 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:41.480694 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.482191 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.484816 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.486245 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.486393 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:41.486609 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:41.487631 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0809 20:28:41.487892 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11253,"thread_start_us":11154,"threads_started":1}
I20260809 20:28:41.491307 31553 tablet_service.cc:1467] Processing CreateTablet for tablet 04e3cdda403548a48f09355960c8daac (DEFAULT_TABLE table=test-table [id=deba7c24ed334875a139a4783a6b352b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:41.491632 31419 tablet_service.cc:1467] Processing CreateTablet for tablet 04e3cdda403548a48f09355960c8daac (DEFAULT_TABLE table=test-table [id=deba7c24ed334875a139a4783a6b352b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:41.492750 31553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e3cdda403548a48f09355960c8daac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:41.493048 31419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e3cdda403548a48f09355960c8daac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:41.500770 31774 tablet_bootstrap.cc:492] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614: Bootstrap starting.
I20260809 20:28:41.504743 31774 tablet_bootstrap.cc:654] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:41.506345 31774 log.cc:824] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:41.506721 31776 tablet_bootstrap.cc:492] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f: Bootstrap starting.
I20260809 20:28:41.509299 31775 tablet_bootstrap.cc:492] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7: Bootstrap starting.
I20260809 20:28:41.510130 31776 tablet_bootstrap.cc:654] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:41.511447 31776 log.cc:824] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:41.512844 31775 tablet_bootstrap.cc:654] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:41.514097 31775 log.cc:824] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:41.514729 31776 tablet_bootstrap.cc:492] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f: No bootstrap required, opened a new log
I20260809 20:28:41.515084 31776 ts_tablet_manager.cc:1397] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 20:28:41.515612 31774 tablet_bootstrap.cc:492] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614: No bootstrap required, opened a new log
I20260809 20:28:41.515959 31774 ts_tablet_manager.cc:1397] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20260809 20:28:41.518505 31775 tablet_bootstrap.cc:492] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7: No bootstrap required, opened a new log
I20260809 20:28:41.518776 31775 ts_tablet_manager.cc:1397] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260809 20:28:41.525564 31775 raft_consensus.cc:348] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd869f501a44f578098dd72efcd0614" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 33363 } } peers { permanent_uuid: "6a6cf69d141d4ab9a9652df25991184f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 36769 } } peers { permanent_uuid: "c42d90936b174bfeadbe526e0154a9b7" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43075 } }
I20260809 20:28:41.525718 31776 raft_consensus.cc:348] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd869f501a44f578098dd72efcd0614" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 33363 } } peers { permanent_uuid: "6a6cf69d141d4ab9a9652df25991184f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 36769 } } peers { permanent_uuid: "c42d90936b174bfeadbe526e0154a9b7" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43075 } }
I20260809 20:28:41.526072 31775 raft_consensus.cc:374] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:41.526186 31776 raft_consensus.cc:374] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:41.526203 31775 raft_consensus.cc:729] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42d90936b174bfeadbe526e0154a9b7, State: Initialized, Role: FOLLOWER
I20260809 20:28:41.526319 31776 raft_consensus.cc:729] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6cf69d141d4ab9a9652df25991184f, State: Initialized, Role: FOLLOWER
I20260809 20:28:41.526623 31774 raft_consensus.cc:348] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd869f501a44f578098dd72efcd0614" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 33363 } } peers { permanent_uuid: "6a6cf69d141d4ab9a9652df25991184f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 36769 } } peers { permanent_uuid: "c42d90936b174bfeadbe526e0154a9b7" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43075 } }
I20260809 20:28:41.526844 31775 consensus_queue.cc:260] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd869f501a44f578098dd72efcd0614" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 33363 } } peers { permanent_uuid: "6a6cf69d141d4ab9a9652df25991184f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 36769 } } peers { permanent_uuid: "c42d90936b174bfeadbe526e0154a9b7" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43075 } }
I20260809 20:28:41.527072 31774 raft_consensus.cc:374] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:41.527158 31774 raft_consensus.cc:729] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd869f501a44f578098dd72efcd0614, State: Initialized, Role: FOLLOWER
I20260809 20:28:41.527055 31776 consensus_queue.cc:260] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd869f501a44f578098dd72efcd0614" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 33363 } } peers { permanent_uuid: "6a6cf69d141d4ab9a9652df25991184f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 36769 } } peers { permanent_uuid: "c42d90936b174bfeadbe526e0154a9b7" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43075 } }
I20260809 20:28:41.528343 31620 heartbeater.cc:503] Master 127.12.78.62:33407 was elected leader, sending a full tablet report...
I20260809 20:28:41.527854 31774 consensus_queue.cc:260] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd869f501a44f578098dd72efcd0614" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 33363 } } peers { permanent_uuid: "6a6cf69d141d4ab9a9652df25991184f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 36769 } } peers { permanent_uuid: "c42d90936b174bfeadbe526e0154a9b7" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43075 } }
I20260809 20:28:41.529204 31775 ts_tablet_manager.cc:1428] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260809 20:28:41.530201 31755 heartbeater.cc:503] Master 127.12.78.62:33407 was elected leader, sending a full tablet report...
I20260809 20:28:41.530843 31774 ts_tablet_manager.cc:1428] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260809 20:28:41.531966 31776 ts_tablet_manager.cc:1428] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
W20260809 20:28:41.654657 31756 tablet.cc:2369] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:28:41.683719 31621 tablet.cc:2369] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:28:41.702996 31486 tablet.cc:2369] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:28:41.905925 31782 raft_consensus.cc:482] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:28:41.906256 31782 raft_consensus.cc:504] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd869f501a44f578098dd72efcd0614" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 33363 } } peers { permanent_uuid: "6a6cf69d141d4ab9a9652df25991184f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 36769 } } peers { permanent_uuid: "c42d90936b174bfeadbe526e0154a9b7" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43075 } }
I20260809 20:28:41.907974 31782 leader_election.cc:288] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a6cf69d141d4ab9a9652df25991184f (127.12.78.1:36769), c42d90936b174bfeadbe526e0154a9b7 (127.12.78.2:43075)
I20260809 20:28:41.915537 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:41.908473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:34063 (local address 127.12.78.1:36769)
0809 20:28:41.908750 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:41.908760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:41.908791 (+ 31us) server_negotiation.cc:407] Connection header received
0809 20:28:41.908884 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:41.908892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:41.909007 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:41.909119 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:41.909585 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.910733 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.913238 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.914480 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.914635 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:41.914750 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:41.915145 (+ 395us) server_negotiation.cc:299] Negotiation successful
0809 20:28:41.915281 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260809 20:28:41.915537 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:41.908797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:45203 (local address 127.12.78.2:43075)
0809 20:28:41.909011 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:41.909016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:41.909241 (+ 225us) server_negotiation.cc:407] Connection header received
0809 20:28:41.909344 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:41.909348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:41.909424 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:41.909554 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:41.910013 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.911063 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.913402 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.914582 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.914691 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:41.914787 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:41.915170 (+ 383us) server_negotiation.cc:299] Negotiation successful
0809 20:28:41.915291 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 20:28:41.915566 31757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:41.908177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:36769 (local address 127.12.78.3:34063)
0809 20:28:41.908348 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:41.908369 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:41.908502 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:41.909173 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:41.909181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:41.909213 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:41.909428 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:41.909436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.910885 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.910889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:41.913049 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:41.913064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.914626 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.914630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:41.914864 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:41.914966 (+ 102us) client_negotiation.cc:769] Sending connection context
0809 20:28:41.915064 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 20:28:41.915215 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":135}
I20260809 20:28:41.915621 31783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:41.908418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:43075 (local address 127.12.78.3:45203)
0809 20:28:41.909113 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:41.909130 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:41.909234 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:41.909565 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:41.909570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:41.909599 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:41.909855 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:41.909862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.911197 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.911201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:41.913229 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:41.913244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:41.914735 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:41.914740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:41.914992 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:41.915033 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 20:28:41.915127 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 20:28:41.915226 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":53,"thread_start_us":298,"threads_started":1}
I20260809 20:28:41.916745 31574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e3cdda403548a48f09355960c8daac" candidate_uuid: "dcd869f501a44f578098dd72efcd0614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42d90936b174bfeadbe526e0154a9b7" is_pre_election: true
I20260809 20:28:41.916741 31439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e3cdda403548a48f09355960c8daac" candidate_uuid: "dcd869f501a44f578098dd72efcd0614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6cf69d141d4ab9a9652df25991184f" is_pre_election: true
I20260809 20:28:41.917230 31439 raft_consensus.cc:2454] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcd869f501a44f578098dd72efcd0614 in term 0.
I20260809 20:28:41.917232 31574 raft_consensus.cc:2454] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcd869f501a44f578098dd72efcd0614 in term 0.
I20260809 20:28:41.917882 31640 leader_election.cc:302] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c42d90936b174bfeadbe526e0154a9b7, dcd869f501a44f578098dd72efcd0614; no voters:
I20260809 20:28:41.918409 31782 raft_consensus.cc:2781] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:28:41.918532 31782 raft_consensus.cc:482] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:28:41.918658 31782 raft_consensus.cc:3037] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:41.921032 31782 raft_consensus.cc:504] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd869f501a44f578098dd72efcd0614" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 33363 } } peers { permanent_uuid: "6a6cf69d141d4ab9a9652df25991184f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 36769 } } peers { permanent_uuid: "c42d90936b174bfeadbe526e0154a9b7" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43075 } }
I20260809 20:28:41.921689 31782 leader_election.cc:288] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [CANDIDATE]: Term 1 election: Requested vote from peers 6a6cf69d141d4ab9a9652df25991184f (127.12.78.1:36769), c42d90936b174bfeadbe526e0154a9b7 (127.12.78.2:43075)
I20260809 20:28:41.922127 31439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e3cdda403548a48f09355960c8daac" candidate_uuid: "dcd869f501a44f578098dd72efcd0614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6cf69d141d4ab9a9652df25991184f"
I20260809 20:28:41.922210 31574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e3cdda403548a48f09355960c8daac" candidate_uuid: "dcd869f501a44f578098dd72efcd0614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42d90936b174bfeadbe526e0154a9b7"
I20260809 20:28:41.922360 31439 raft_consensus.cc:3037] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:41.922401 31574 raft_consensus.cc:3037] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:41.925717 31574 raft_consensus.cc:2454] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcd869f501a44f578098dd72efcd0614 in term 1.
I20260809 20:28:41.925778 31439 raft_consensus.cc:2454] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcd869f501a44f578098dd72efcd0614 in term 1.
I20260809 20:28:41.926262 31639 leader_election.cc:302] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a6cf69d141d4ab9a9652df25991184f, dcd869f501a44f578098dd72efcd0614; no voters:
I20260809 20:28:41.926671 31782 raft_consensus.cc:2781] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:28:41.927137 31782 raft_consensus.cc:686] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [term 1 LEADER]: Becoming Leader. State: Replica: dcd869f501a44f578098dd72efcd0614, State: Running, Role: LEADER
I20260809 20:28:41.927632 31782 consensus_queue.cc:237] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd869f501a44f578098dd72efcd0614" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 33363 } } peers { permanent_uuid: "6a6cf69d141d4ab9a9652df25991184f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 36769 } } peers { permanent_uuid: "c42d90936b174bfeadbe526e0154a9b7" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43075 } }
I20260809 20:28:41.932901 31293 catalog_manager.cc:5495] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcd869f501a44f578098dd72efcd0614 (127.12.78.3). New cstate: current_term: 1 leader_uuid: "dcd869f501a44f578098dd72efcd0614" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd869f501a44f578098dd72efcd0614" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 33363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6cf69d141d4ab9a9652df25991184f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 36769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c42d90936b174bfeadbe526e0154a9b7" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43075 } health_report { overall_health: UNKNOWN } } }
I20260809 20:28:42.052295 31787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:42.046426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50910 (local address 127.12.78.3:33363)
0809 20:28:42.046927 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:42.046933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:42.047090 (+ 157us) server_negotiation.cc:407] Connection header received
0809 20:28:42.047255 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:42.047259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:42.047338 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:42.047559 (+ 221us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:28:42.047963 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:42.048760 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:42.050220 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:42.050648 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:42.050751 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:42.051151 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:28:42.051693 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:28:42.051805 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:42.051878 (+ 73us) server_negotiation.cc:299] Negotiation successful
0809 20:28:42.051942 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":252,"threads_started":1}
I20260809 20:28:42.060138 31439 raft_consensus.cc:1261] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f [term 1 FOLLOWER]: Refusing update from remote peer dcd869f501a44f578098dd72efcd0614: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:28:42.060137 31574 raft_consensus.cc:1261] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7 [term 1 FOLLOWER]: Refusing update from remote peer dcd869f501a44f578098dd72efcd0614: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:28:42.061182 31784 consensus_queue.cc:1035] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6cf69d141d4ab9a9652df25991184f" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 36769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:28:42.061641 31782 consensus_queue.cc:1035] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42d90936b174bfeadbe526e0154a9b7" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 43075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:28:42.072976 31789 mvcc.cc:205] Tried to move back new op lower bound from 7316714791148830720 to 7316714790619664384. Current Snapshot: MvccSnapshot[applied={T|T < 7316714791148830720}]
I20260809 20:28:42.074337 31794 mvcc.cc:205] Tried to move back new op lower bound from 7316714791148830720 to 7316714790619664384. Current Snapshot: MvccSnapshot[applied={T|T < 7316714791148830720}]
I20260809 20:28:42.088536 31292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:28:42.088840 31292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:28:42.092458 31292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4862e8d746b64503a15705979aa34db2: Sending DeleteTablet for 3 replicas of tablet 04e3cdda403548a48f09355960c8daac
I20260809 20:28:42.093773 31419 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e3cdda403548a48f09355960c8daac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:42 UTC) from {username='slave'} at 127.0.0.1:57864
I20260809 20:28:42.093983 31553 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e3cdda403548a48f09355960c8daac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:42 UTC) from {username='slave'} at 127.0.0.1:60278
I20260809 20:28:42.094485 31687 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e3cdda403548a48f09355960c8daac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:42 UTC) from {username='slave'} at 127.0.0.1:50904
I20260809 20:28:42.094851 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 31353
I20260809 20:28:42.104940 31802 tablet_replica.cc:330] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f: stopping tablet replica
I20260809 20:28:42.106287 31802 raft_consensus.cc:2229] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:28:42.107710 31802 raft_consensus.cc:2258] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:42.108506 31803 tablet_replica.cc:330] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7: stopping tablet replica
I20260809 20:28:42.109438 31803 raft_consensus.cc:2229] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:28:42.109934 31803 raft_consensus.cc:2258] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:42.111678 31802 ts_tablet_manager.cc:1905] T 04e3cdda403548a48f09355960c8daac P 6a6cf69d141d4ab9a9652df25991184f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:42.112703 31804 tablet_replica.cc:330] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614: stopping tablet replica
I20260809 20:28:42.113068 31803 ts_tablet_manager.cc:1905] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:42.113103 31804 raft_consensus.cc:2229] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:28:42.113858 31804 raft_consensus.cc:2258] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:42.115618 31804 ts_tablet_manager.cc:1905] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:42.118220 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:42.096738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.12.78.1:36769)
0809 20:28:42.096973 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:42.096982 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:42.107229 (+ 10247us) server_negotiation.cc:407] Connection header received
0809 20:28:42.107739 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:42.107746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:42.107862 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:42.107961 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:42.108771 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:42.109870 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:42.114240 (+ 4370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:42.114886 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:42.114987 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:42.116485 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:42.116496 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:42.116512 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:42.117291 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:42.117351 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:42.117373 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:42.117472 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:42.117720 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 20:28:42.117846 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 20:28:42.119620 31480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:42.123224 31804 ts_tablet_manager.cc:1918] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:42.123477 31804 log.cc:1196] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/04e3cdda403548a48f09355960c8daac
I20260809 20:28:42.124179 31804 ts_tablet_manager.cc:1939] T 04e3cdda403548a48f09355960c8daac P dcd869f501a44f578098dd72efcd0614: Deleting consensus metadata
I20260809 20:28:42.126313 31279 catalog_manager.cc:4841] TS dcd869f501a44f578098dd72efcd0614 (127.12.78.3:33363): tablet 04e3cdda403548a48f09355960c8daac (table test-table [id=deba7c24ed334875a139a4783a6b352b]) successfully deleted
I20260809 20:28:42.127185 31803 ts_tablet_manager.cc:1918] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:42.127400 31803 log.cc:1196] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/04e3cdda403548a48f09355960c8daac
I20260809 20:28:42.128039 31803 ts_tablet_manager.cc:1939] T 04e3cdda403548a48f09355960c8daac P c42d90936b174bfeadbe526e0154a9b7: Deleting consensus metadata
I20260809 20:28:42.130779 31279 catalog_manager.cc:4841] TS c42d90936b174bfeadbe526e0154a9b7 (127.12.78.2:43075): tablet 04e3cdda403548a48f09355960c8daac (table test-table [id=deba7c24ed334875a139a4783a6b352b]) successfully deleted
I20260809 20:28:42.218443 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 31353
I20260809 20:28:42.238538 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 31489
W20260809 20:28:42.238503 31278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.78.1:36769 (error 108)
I20260809 20:28:42.241273 31808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:42.239991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:36769 (local address 127.0.0.1:57882)
0809 20:28:42.240849 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:42.241009 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":589,"threads_started":1}
W20260809 20:28:42.241568 31278 catalog_manager.cc:4584] TS 6a6cf69d141d4ab9a9652df25991184f (127.12.78.1:36769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04e3cdda403548a48f09355960c8daac: Network error: Client connection negotiation failed: client connection to 127.12.78.1:36769: connect: Connection refused (error 111)
I20260809 20:28:42.247274 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:42.239684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.12.78.2:43075)
0809 20:28:42.239870 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:42.239876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:42.239919 (+ 43us) server_negotiation.cc:407] Connection header received
0809 20:28:42.239992 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:42.239997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:42.240121 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:42.240256 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:42.240902 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:42.242078 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:42.244064 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:42.244468 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:42.244750 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:42.245849 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:42.245857 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:42.245871 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:42.246457 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:42.246509 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:42.246529 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:42.246702 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:42.246847 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 20:28:42.246935 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 20:28:42.247924 31615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:42.295410 31808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:42.294842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:36769 (local address 127.0.0.1:57896)
0809 20:28:42.295107 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:42.295227 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260809 20:28:42.296651 31808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:42.296198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:36769 (local address 127.0.0.1:57912)
0809 20:28:42.296362 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:42.296451 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 20:28:42.339530 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 31489
I20260809 20:28:42.357429 31808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:42.356837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:36769 (local address 127.0.0.1:57928)
0809 20:28:42.357091 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:42.357218 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260809 20:28:42.358803 31808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:42.358338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:36769 (local address 127.0.0.1:57940)
0809 20:28:42.358522 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:42.358618 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 20:28:42.360406 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 31623
I20260809 20:28:42.368350 31787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:42.361294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.12.78.3:33363)
0809 20:28:42.361453 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:42.361458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:42.361475 (+ 17us) server_negotiation.cc:407] Connection header received
0809 20:28:42.361563 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:42.361566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:42.361649 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:42.361734 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:42.362342 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:42.363126 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:42.365093 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:42.365570 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:42.365677 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:42.366862 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:42.366869 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:42.366884 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:42.367578 (+ 694us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:42.367625 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:42.367646 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:42.367736 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:42.367911 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 20:28:42.367989 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 20:28:42.369140 31750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:42.439499 31808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:42.438879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:36769 (local address 127.0.0.1:57946)
0809 20:28:42.439137 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:42.439253 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 20:28:42.440702 31808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:42.440224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:36769 (local address 127.0.0.1:57952)
0809 20:28:42.440426 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:42.440517 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260809 20:28:42.492730 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 31623
I20260809 20:28:42.515543 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 31261
I20260809 20:28:42.516705 31324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:42.624898 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 31261
I20260809 20:28:42.625023 31808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:42.624115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:36769 (local address 127.0.0.1:57960)
0809 20:28:42.624732 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:42.624833 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436}
2026-08-09T20:28:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3265 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:28:42 dist-test-slave-rnrw krb5kdc[31814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:28:42 dist-test-slave-rnrw krb5kdc[31814](info): set up 2 sockets
Aug 09 20:28:42 dist-test-slave-rnrw krb5kdc[31814](info): commencing operation
krb5kdc: starting...
W20260809 20:28:43.811923 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:28:43 dist-test-slave-rnrw krb5kdc[31814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:28:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:28:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:28:44.100375 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:35913
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:35913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:44.442001 31827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:44.442309 31827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:44.442399 31827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:44.449527 31827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:28:44.449627 31827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:44.449674 31827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:44.449733 31827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:28:44.449777 31827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:28:44.459334 31827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:35913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:35913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:44.460647 31827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:44.461913 31827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:44.473563 31834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:44.473769 31836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:44.474061 31833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:44.474529 31827 server_base.cc:1034] running on GCE node
I20260809 20:28:44.475363 31827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:44.476810 31827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:44.478044 31827 hybrid_clock.cc:648] HybridClock initialized: now 1786307324477995 us; error 49 us; skew 500 ppm
I20260809 20:28:44.478415 31827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:44 dist-test-slave-rnrw krb5kdc[31814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307324, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:44.481504 31827 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:28:44.482918 31827 webserver.cc:460] Webserver started at http://127.12.78.62:41207/ using document root <none> and password file <none>
I20260809 20:28:44.483613 31827 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:44.483759 31827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:44.484066 31827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:44.487095 31827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "520039dadf9a4112a858b55a06f6ba82"
format_stamp: "Formatted at 2026-08-09 20:28:44 on dist-test-slave-rnrw"
server_key: "cf07000eb6b306bf5be0e47443c48bc3"
server_key_iv: "3f0f3983c0b910ddac30935bb480138b"
server_key_version: "encryptionkey@0"
I20260809 20:28:44.487928 31827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "520039dadf9a4112a858b55a06f6ba82"
format_stamp: "Formatted at 2026-08-09 20:28:44 on dist-test-slave-rnrw"
server_key: "cf07000eb6b306bf5be0e47443c48bc3"
server_key_iv: "3f0f3983c0b910ddac30935bb480138b"
server_key_version: "encryptionkey@0"
I20260809 20:28:44.493140 31827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260809 20:28:44.496600 31843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:44.497923 31827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:28:44.498131 31827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "520039dadf9a4112a858b55a06f6ba82"
format_stamp: "Formatted at 2026-08-09 20:28:44 on dist-test-slave-rnrw"
server_key: "cf07000eb6b306bf5be0e47443c48bc3"
server_key_iv: "3f0f3983c0b910ddac30935bb480138b"
server_key_version: "encryptionkey@0"
I20260809 20:28:44.498337 31827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:44.512135 31827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:44.515967 31827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:44.516258 31827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:44.535105 31827 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:35913
I20260809 20:28:44.535128 31896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:35913 every 8 connection(s)
I20260809 20:28:44.536527 31827 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:28:44.538875 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 31827
I20260809 20:28:44.539099 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:28:44.539556 12600 external_mini_cluster.cc:1428] Setting key e52d2a249c992c9571cace5e69eea1e9
I20260809 20:28:44.540968 31897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 20:28:44 dist-test-slave-rnrw krb5kdc[31814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:28:44.556589 31897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82: Bootstrap starting.
I20260809 20:28:44.561540 31897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:44.562901 31897 log.cc:824] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:44.566816 31897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82: No bootstrap required, opened a new log
I20260809 20:28:44.566859 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:44.541512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56008 (local address 127.12.78.62:35913)
0809 20:28:44.542297 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:44.542308 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:44.542349 (+ 41us) server_negotiation.cc:407] Connection header received
0809 20:28:44.543972 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:44.543993 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:44.544374 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:44.544791 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:44.546758 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:44.548177 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:44.550625 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:44.551314 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:44.551460 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:44.556994 (+ 5534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:44.557005 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:44.557028 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:44.557083 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:44.561600 (+ 4517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:44.562389 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:44.562398 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:44.562407 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:44.562668 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:44.563084 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:44.563090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:44.563093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:44.563888 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:44.564150 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:44.564898 (+ 748us) server_negotiation.cc:299] Negotiation successful
0809 20:28:44.565375 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":351,"threads_started":1}
I20260809 20:28:44.575907 31897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520039dadf9a4112a858b55a06f6ba82" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35913 } }
I20260809 20:28:44.576297 31897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:44.576380 31897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 520039dadf9a4112a858b55a06f6ba82, State: Initialized, Role: FOLLOWER
I20260809 20:28:44.577054 31897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520039dadf9a4112a858b55a06f6ba82" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35913 } }
I20260809 20:28:44.577276 31897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:28:44.577385 31897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:28:44.577534 31897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:44.580178 31897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520039dadf9a4112a858b55a06f6ba82" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35913 } }
I20260809 20:28:44.580744 31897 leader_election.cc:302] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 520039dadf9a4112a858b55a06f6ba82; no voters:
I20260809 20:28:44.581393 31897 leader_election.cc:288] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:28:44.581594 31904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:28:44.582382 31904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [term 1 LEADER]: Becoming Leader. State: Replica: 520039dadf9a4112a858b55a06f6ba82, State: Running, Role: LEADER
I20260809 20:28:44.583016 31904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520039dadf9a4112a858b55a06f6ba82" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35913 } }
I20260809 20:28:44.583655 31897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:28:44.585800 31905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "520039dadf9a4112a858b55a06f6ba82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520039dadf9a4112a858b55a06f6ba82" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35913 } } }
I20260809 20:28:44.586174 31905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:44.586434 31906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 520039dadf9a4112a858b55a06f6ba82. Latest consensus state: current_term: 1 leader_uuid: "520039dadf9a4112a858b55a06f6ba82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520039dadf9a4112a858b55a06f6ba82" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35913 } } }
I20260809 20:28:44.586879 31906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:44.589649 31909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:28:44.595414 31909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:28:44.605525 31909 catalog_manager.cc:1347] Generated new cluster ID: dba3bab96e1941f99142acf4a3fe2e9c
I20260809 20:28:44.605651 31909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:28:44.626556 31909 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:28:44.628072 31909 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:28:44.640790 31909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82: Generated new TSK 0
I20260809 20:28:44.641611 31909 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:28:44.727360 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:35913
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:45.147248 31925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:45.147543 31925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:45.147708 31925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:45.154889 31925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:45.155018 31925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:45.155211 31925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:28:45.164904 31925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:35913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:45.166221 31925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:45.167551 31925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:45.180927 31931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:45.181070 31930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:45.181032 31933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:45.181728 31925 server_base.cc:1034] running on GCE node
I20260809 20:28:45.182281 31925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:45.183130 31925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:45.184329 31925 hybrid_clock.cc:648] HybridClock initialized: now 1786307325184291 us; error 31 us; skew 500 ppm
I20260809 20:28:45.184656 31925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:45 dist-test-slave-rnrw krb5kdc[31814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307325, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:45.187844 31925 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:28:45.189412 31925 webserver.cc:460] Webserver started at http://127.12.78.1:38267/ using document root <none> and password file <none>
I20260809 20:28:45.190158 31925 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:45.190286 31925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:45.190620 31925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:45.193658 31925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "74ccf746e3dd4724a6cae38fd0c2fd20"
format_stamp: "Formatted at 2026-08-09 20:28:45 on dist-test-slave-rnrw"
server_key: "3d4893b25889da0ccf33883f4e6ded94"
server_key_iv: "a6e880b5b7152d0bcfaa6d6fc9e8aff5"
server_key_version: "encryptionkey@0"
I20260809 20:28:45.194420 31925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "74ccf746e3dd4724a6cae38fd0c2fd20"
format_stamp: "Formatted at 2026-08-09 20:28:45 on dist-test-slave-rnrw"
server_key: "3d4893b25889da0ccf33883f4e6ded94"
server_key_iv: "a6e880b5b7152d0bcfaa6d6fc9e8aff5"
server_key_version: "encryptionkey@0"
I20260809 20:28:45.199604 31925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:28:45.203238 31940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:45.204522 31925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 20:28:45.204763 31925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "74ccf746e3dd4724a6cae38fd0c2fd20"
format_stamp: "Formatted at 2026-08-09 20:28:45 on dist-test-slave-rnrw"
server_key: "3d4893b25889da0ccf33883f4e6ded94"
server_key_iv: "a6e880b5b7152d0bcfaa6d6fc9e8aff5"
server_key_version: "encryptionkey@0"
I20260809 20:28:45.204962 31925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:45.220366 31925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:45.224850 31925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:45.225157 31925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:45.226197 31925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:45.227758 31925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:45.227836 31925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:45.227986 31925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:45.228044 31925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:45.262504 31925 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:43081
I20260809 20:28:45.262539 32058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:43081 every 8 connection(s)
I20260809 20:28:45.263912 31925 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:28:45.272939 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 31925
I20260809 20:28:45.273162 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:28:45.273602 12600 external_mini_cluster.cc:1428] Setting key 1762b99872a3f026e519a2156447c7be
Aug 09 20:28:45 dist-test-slave-rnrw krb5kdc[31814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307325, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:28:45.284891 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:45.266580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:35493 (local address 127.12.78.62:35913)
0809 20:28:45.267064 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:45.267069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:45.267918 (+ 849us) server_negotiation.cc:407] Connection header received
0809 20:28:45.270258 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:45.270264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:45.270358 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:45.270476 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:45.272042 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:45.272842 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:45.274633 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:45.275344 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:45.275483 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:45.279058 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:45.279063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:45.279066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:45.279101 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:45.281667 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:45.282409 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:45.282415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:45.282418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:45.282548 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:45.282999 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:45.283005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:45.283008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:45.283298 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:45.283476 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:45.284190 (+ 714us) server_negotiation.cc:299] Negotiation successful
0809 20:28:45.284464 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":367,"thread_start_us":261,"threads_started":1}
I20260809 20:28:45.285913 32063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:45.267015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:35913 (local address 127.12.78.1:35493)
0809 20:28:45.267759 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:45.267797 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:45.269968 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:45.270734 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:45.270742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:45.271288 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:45.271860 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:45.271877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:45.273002 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:45.273007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:45.274446 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:45.274461 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:45.275610 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:45.275619 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:45.275874 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:45.276609 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:28:45.276641 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:45.278833 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:45.281821 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:45.281830 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:45.281848 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:45.282286 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:45.282669 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:45.282674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:45.282678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:45.282887 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:45.283427 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:45.283435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:45.283820 (+ 385us) client_negotiation.cc:769] Sending connection context
0809 20:28:45.284124 (+ 304us) client_negotiation.cc:240] Negotiation successful
0809 20:28:45.284703 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":276,"threads_started":1}
I20260809 20:28:45.287531 32059 heartbeater.cc:347] Connected to a master server at 127.12.78.62:35913
I20260809 20:28:45.287873 32059 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:45.288774 32059 heartbeater.cc:511] Master 127.12.78.62:35913 requested a full tablet report, sending...
I20260809 20:28:45.291180 31860 ts_manager.cc:194] Registered new tserver with Master: 74ccf746e3dd4724a6cae38fd0c2fd20 (127.12.78.1:43081)
I20260809 20:28:45.293372 31860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:35493
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:28:45.345077 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:35913
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:45.694023 32066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:45.694303 32066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:45.694396 32066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:45.701546 32066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:45.701656 32066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:45.701841 32066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:28:45.711653 32066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:35913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:45.712962 32066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:45.714320 32066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:45.727488 32072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:45.727384 32074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:45.727524 32071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:45.729813 32066 server_base.cc:1034] running on GCE node
I20260809 20:28:45.730423 32066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:45.731369 32066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:45.732637 32066 hybrid_clock.cc:648] HybridClock initialized: now 1786307325732594 us; error 51 us; skew 500 ppm
I20260809 20:28:45.732986 32066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:45 dist-test-slave-rnrw krb5kdc[31814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307325, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:45.736132 32066 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:28:45.737681 32066 webserver.cc:460] Webserver started at http://127.12.78.2:34175/ using document root <none> and password file <none>
I20260809 20:28:45.738356 32066 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:45.738502 32066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:45.738806 32066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:45.741673 32066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "5d2a17c7e61f4faaa8f503f2670153b8"
format_stamp: "Formatted at 2026-08-09 20:28:45 on dist-test-slave-rnrw"
server_key: "9db8c23c9dcafdff95aa452d0eaf2cdb"
server_key_iv: "36beb33ba34826eb9d47e2741a23c63c"
server_key_version: "encryptionkey@0"
I20260809 20:28:45.742422 32066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "5d2a17c7e61f4faaa8f503f2670153b8"
format_stamp: "Formatted at 2026-08-09 20:28:45 on dist-test-slave-rnrw"
server_key: "9db8c23c9dcafdff95aa452d0eaf2cdb"
server_key_iv: "36beb33ba34826eb9d47e2741a23c63c"
server_key_version: "encryptionkey@0"
I20260809 20:28:45.747593 32066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 20:28:45.751315 32081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:45.752713 32066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:28:45.752894 32066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "5d2a17c7e61f4faaa8f503f2670153b8"
format_stamp: "Formatted at 2026-08-09 20:28:45 on dist-test-slave-rnrw"
server_key: "9db8c23c9dcafdff95aa452d0eaf2cdb"
server_key_iv: "36beb33ba34826eb9d47e2741a23c63c"
server_key_version: "encryptionkey@0"
I20260809 20:28:45.753104 32066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:45.778383 32066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:45.782718 32066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:45.783031 32066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:45.784111 32066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:45.785645 32066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:45.785754 32066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:45.785912 32066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:45.785988 32066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:45.821621 32066 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:42819
I20260809 20:28:45.821658 32200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:42819 every 8 connection(s)
I20260809 20:28:45.823005 32066 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:28:45.826272 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 32066
I20260809 20:28:45.826500 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:28:45.827203 12600 external_mini_cluster.cc:1428] Setting key b792e816b7e0d7d5bf806f07248506f1
Aug 09 20:28:45 dist-test-slave-rnrw krb5kdc[31814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307325, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:28:45.844023 32203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:45.825644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:58945 (local address 127.12.78.62:35913)
0809 20:28:45.826124 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:45.826129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:45.827196 (+ 1067us) server_negotiation.cc:407] Connection header received
0809 20:28:45.829420 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:45.829425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:45.829508 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:45.829633 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:45.831137 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:45.831846 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:45.833437 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:45.833883 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:45.834032 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:45.837638 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:45.837647 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:45.837655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:45.837704 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:45.840959 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:45.841687 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:45.841698 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:45.841701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:45.841837 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:45.842290 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:45.842295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:45.842297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:45.842560 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:45.842717 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:45.843388 (+ 671us) server_negotiation.cc:299] Negotiation successful
0809 20:28:45.843640 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":262,"threads_started":1}
I20260809 20:28:45.845103 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:45.826203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:35913 (local address 127.12.78.2:58945)
0809 20:28:45.827036 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:45.827080 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:45.829170 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:45.829870 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:45.829878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:45.830414 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:45.830963 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:45.830981 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:45.831990 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:45.831995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:45.833289 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:45.833301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:45.834100 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:45.834107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:45.834345 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:45.835005 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:28:45.835027 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:45.837396 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:45.841099 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:45.841108 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:45.841124 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:45.841556 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:45.841960 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:45.841965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:45.841968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:45.842164 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:45.842689 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:45.842697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:45.843029 (+ 332us) client_negotiation.cc:769] Sending connection context
0809 20:28:45.843290 (+ 261us) client_negotiation.cc:240] Negotiation successful
0809 20:28:45.843817 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":377,"threads_started":1}
I20260809 20:28:45.846666 32201 heartbeater.cc:347] Connected to a master server at 127.12.78.62:35913
I20260809 20:28:45.847028 32201 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:45.847872 32201 heartbeater.cc:511] Master 127.12.78.62:35913 requested a full tablet report, sending...
I20260809 20:28:45.849606 31860 ts_manager.cc:194] Registered new tserver with Master: 5d2a17c7e61f4faaa8f503f2670153b8 (127.12.78.2:42819)
I20260809 20:28:45.850517 31860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:58945
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:28:45.891673 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:35913
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:46.224997 32208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:46.225329 32208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:46.225494 32208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:46.232833 32208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:46.232981 32208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:46.233166 32208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:28:46.243413 32208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:35913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:46.244906 32208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:46.246260 32208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:46.260201 32213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:46.260221 32214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:46.261006 32216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:46.261042 32208 server_base.cc:1034] running on GCE node
I20260809 20:28:46.261780 32208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:46.262979 32208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:46.264230 32208 hybrid_clock.cc:648] HybridClock initialized: now 1786307326264140 us; error 80 us; skew 500 ppm
I20260809 20:28:46.264631 32208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:46 dist-test-slave-rnrw krb5kdc[31814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307326, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:46.268703 32208 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:28:46.270277 32208 webserver.cc:460] Webserver started at http://127.12.78.3:38363/ using document root <none> and password file <none>
I20260809 20:28:46.270959 32208 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:46.271145 32208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:46.271495 32208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:46.274462 32208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "5b77093b4b8a4fd489210bffb62f97b8"
format_stamp: "Formatted at 2026-08-09 20:28:46 on dist-test-slave-rnrw"
server_key: "20ec2b370094ad0d72e6ecf7c7f2f40c"
server_key_iv: "95ef631b83a7b771a462fe90e9d337fc"
server_key_version: "encryptionkey@0"
I20260809 20:28:46.275234 32208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "5b77093b4b8a4fd489210bffb62f97b8"
format_stamp: "Formatted at 2026-08-09 20:28:46 on dist-test-slave-rnrw"
server_key: "20ec2b370094ad0d72e6ecf7c7f2f40c"
server_key_iv: "95ef631b83a7b771a462fe90e9d337fc"
server_key_version: "encryptionkey@0"
I20260809 20:28:46.280678 32208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 20:28:46.284237 32223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:46.285638 32208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260809 20:28:46.285817 32208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "5b77093b4b8a4fd489210bffb62f97b8"
format_stamp: "Formatted at 2026-08-09 20:28:46 on dist-test-slave-rnrw"
server_key: "20ec2b370094ad0d72e6ecf7c7f2f40c"
server_key_iv: "95ef631b83a7b771a462fe90e9d337fc"
server_key_version: "encryptionkey@0"
I20260809 20:28:46.286017 32208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:46.297463 32059 heartbeater.cc:503] Master 127.12.78.62:35913 was elected leader, sending a full tablet report...
I20260809 20:28:46.300920 32208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:46.305649 32208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:46.305948 32208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:46.306892 32208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:46.308254 32208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:46.308353 32208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:46.308522 32208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:46.308656 32208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260809 20:28:46.341092 32208 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:43821
I20260809 20:28:46.341138 32344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:43821 every 8 connection(s)
I20260809 20:28:46.342370 32208 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:28:46.349489 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 32208
I20260809 20:28:46.349696 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:28:46.350287 12600 external_mini_cluster.cc:1428] Setting key 0ac6011d2abe872758ccc6ddedd8de26
Aug 09 20:28:46 dist-test-slave-rnrw krb5kdc[31814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307326, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:28:46.362272 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.344947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:41023 (local address 127.12.78.62:35913)
0809 20:28:46.345409 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:46.345414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:46.346336 (+ 922us) server_negotiation.cc:407] Connection header received
0809 20:28:46.348097 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:46.348102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:46.348203 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:46.348319 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:46.350146 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.350986 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.353010 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.353516 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.353623 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.356916 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:46.356921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:46.356925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:46.356958 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:46.359654 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:46.360293 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:46.360297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:46.360299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:46.360413 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:46.360824 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:46.360827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:46.360829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:46.361102 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:46.361220 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:46.361809 (+ 589us) server_negotiation.cc:299] Negotiation successful
0809 20:28:46.361978 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":245,"threads_started":1}
I20260809 20:28:46.363132 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.345383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:35913 (local address 127.12.78.3:41023)
0809 20:28:46.346180 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:46.346216 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:46.347836 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:46.348580 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:46.348595 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:46.349150 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:46.349949 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:46.349975 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.351150 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.351156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:46.352862 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:46.352881 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.353703 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.353708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:46.353891 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.354493 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:28:46.354513 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:46.356661 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:46.359769 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:46.359775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:46.359787 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:46.360188 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:46.360499 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:46.360502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:46.360504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:46.360724 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:46.361201 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:46.361207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:46.361500 (+ 293us) client_negotiation.cc:769] Sending connection context
0809 20:28:46.361722 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 20:28:46.362077 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":362,"threads_started":1}
I20260809 20:28:46.364629 32345 heartbeater.cc:347] Connected to a master server at 127.12.78.62:35913
I20260809 20:28:46.364951 32345 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:46.365728 32345 heartbeater.cc:511] Master 127.12.78.62:35913 requested a full tablet report, sending...
I20260809 20:28:46.367146 31859 ts_manager.cc:194] Registered new tserver with Master: 5b77093b4b8a4fd489210bffb62f97b8 (127.12.78.3:43821)
I20260809 20:28:46.368016 31859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:41023
I20260809 20:28:46.377641 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:28:46.395195 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.381571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.12.78.62:35913)
0809 20:28:46.381779 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:46.381786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:46.382223 (+ 437us) server_negotiation.cc:407] Connection header received
0809 20:28:46.382414 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:46.382423 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:46.382486 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:46.382583 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:46.383995 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.384802 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.386182 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.386552 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.386645 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.389324 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:46.389328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:46.389332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:46.389363 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:46.392133 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:46.393021 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:46.393026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:46.393030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:46.393175 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:46.393630 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:46.393635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:46.393639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:46.393949 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:46.394258 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:46.394621 (+ 363us) server_negotiation.cc:299] Negotiation successful
0809 20:28:46.394826 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 20:28:46.399272 31859 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:28:46.406435 31859 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:28:46.428996 32361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.418567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.12.78.2:42819)
0809 20:28:46.420073 (+ 1506us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:46.420082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:46.420117 (+ 35us) server_negotiation.cc:407] Connection header received
0809 20:28:46.420235 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:46.420243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:46.420499 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:46.420744 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:46.421528 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.423101 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.426054 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.427583 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.427685 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.427876 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:46.428456 (+ 580us) server_negotiation.cc:299] Negotiation successful
0809 20:28:46.428666 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1363,"thread_start_us":1277,"threads_started":1}
I20260809 20:28:46.429037 32360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.418292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:42819 (local address 127.0.0.1:57508)
0809 20:28:46.419013 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:46.419077 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:46.419270 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:46.420780 (+ 1510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:46.420787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:46.420819 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:46.421317 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:46.421331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.423297 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.423302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:46.425854 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:46.425874 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.427943 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.427949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:46.428209 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.428266 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 20:28:46.428394 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 20:28:46.428620 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":4,"thread_start_us":1516,"threads_started":1}
I20260809 20:28:46.430970 32362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.419182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:43821 (local address 127.0.0.1:41442)
0809 20:28:46.420460 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:46.420478 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:46.420620 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:46.423697 (+ 3077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:46.423704 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:46.423734 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:46.424296 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:46.424305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.425657 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.425664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:46.427625 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:46.427637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.430036 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.430042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:46.430239 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.430275 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 20:28:46.430369 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 20:28:46.430514 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1165,"mutex_wait_us":25,"thread_start_us":995,"threads_started":1}
I20260809 20:28:46.431118 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.420677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:43081 (local address 127.0.0.1:33560)
0809 20:28:46.421375 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:46.421393 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:46.421496 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:46.423361 (+ 1865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:46.423365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:46.423393 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:46.424012 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:46.424024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.425977 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.425984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:46.428211 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:46.428223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.430228 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.430235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:46.430418 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.430450 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 20:28:46.430516 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 20:28:46.430658 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":105,"thread_start_us":905,"threads_started":1}
I20260809 20:28:46.431957 32365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.421401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.12.78.1:43081)
0809 20:28:46.422179 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:46.422188 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:46.422215 (+ 27us) server_negotiation.cc:407] Connection header received
0809 20:28:46.422308 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:46.422315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:46.422522 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:46.423366 (+ 844us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:46.424231 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.425813 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.428357 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.429946 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.431365 (+ 1419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.431537 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:46.431593 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 20:28:46.431726 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":1676,"threads_started":1}
I20260809 20:28:46.431996 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.419826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.12.78.3:43821)
0809 20:28:46.423115 (+ 3289us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:46.423123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:46.423153 (+ 30us) server_negotiation.cc:407] Connection header received
0809 20:28:46.423246 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:46.423254 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:46.423469 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:46.423668 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:46.424454 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.425511 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.427779 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.429504 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.429658 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.429879 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:46.431517 (+ 1638us) server_negotiation.cc:299] Negotiation successful
0809 20:28:46.431707 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3130,"thread_start_us":2998,"threads_started":1}
I20260809 20:28:46.433650 32131 tablet_service.cc:1467] Processing CreateTablet for tablet 268b6db41f6d4d9f8f075e6c990d6c50 (DEFAULT_TABLE table=test-table [id=222e7dd1489d48aa8346076751a8882a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:46.435336 32131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 268b6db41f6d4d9f8f075e6c990d6c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:46.435022 31991 tablet_service.cc:1467] Processing CreateTablet for tablet 268b6db41f6d4d9f8f075e6c990d6c50 (DEFAULT_TABLE table=test-table [id=222e7dd1489d48aa8346076751a8882a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:46.435482 32273 tablet_service.cc:1467] Processing CreateTablet for tablet 268b6db41f6d4d9f8f075e6c990d6c50 (DEFAULT_TABLE table=test-table [id=222e7dd1489d48aa8346076751a8882a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:46.436602 31991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 268b6db41f6d4d9f8f075e6c990d6c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:46.436952 32273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 268b6db41f6d4d9f8f075e6c990d6c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:46.447605 32366 tablet_bootstrap.cc:492] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20: Bootstrap starting.
I20260809 20:28:46.450497 32367 tablet_bootstrap.cc:492] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8: Bootstrap starting.
I20260809 20:28:46.452112 32368 tablet_bootstrap.cc:492] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8: Bootstrap starting.
I20260809 20:28:46.453172 32366 tablet_bootstrap.cc:654] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:46.453974 32367 tablet_bootstrap.cc:654] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:46.454510 32366 log.cc:824] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:46.455222 32367 log.cc:824] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:46.455495 32368 tablet_bootstrap.cc:654] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:46.456591 32368 log.cc:824] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:46.457690 32366 tablet_bootstrap.cc:492] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20: No bootstrap required, opened a new log
I20260809 20:28:46.457968 32366 ts_tablet_manager.cc:1397] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260809 20:28:46.459024 32367 tablet_bootstrap.cc:492] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8: No bootstrap required, opened a new log
I20260809 20:28:46.459291 32367 ts_tablet_manager.cc:1397] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 20:28:46.459347 32368 tablet_bootstrap.cc:492] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8: No bootstrap required, opened a new log
I20260809 20:28:46.459594 32368 ts_tablet_manager.cc:1397] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260809 20:28:46.468089 32368 raft_consensus.cc:348] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2a17c7e61f4faaa8f503f2670153b8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42819 } } peers { permanent_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 43081 } } peers { permanent_uuid: "5b77093b4b8a4fd489210bffb62f97b8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43821 } }
I20260809 20:28:46.468534 32368 raft_consensus.cc:374] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:46.468675 32368 raft_consensus.cc:729] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b77093b4b8a4fd489210bffb62f97b8, State: Initialized, Role: FOLLOWER
I20260809 20:28:46.469158 32366 raft_consensus.cc:348] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2a17c7e61f4faaa8f503f2670153b8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42819 } } peers { permanent_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 43081 } } peers { permanent_uuid: "5b77093b4b8a4fd489210bffb62f97b8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43821 } }
I20260809 20:28:46.469363 32368 consensus_queue.cc:260] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2a17c7e61f4faaa8f503f2670153b8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42819 } } peers { permanent_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 43081 } } peers { permanent_uuid: "5b77093b4b8a4fd489210bffb62f97b8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43821 } }
I20260809 20:28:46.469583 32366 raft_consensus.cc:374] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:46.469758 32366 raft_consensus.cc:729] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ccf746e3dd4724a6cae38fd0c2fd20, State: Initialized, Role: FOLLOWER
I20260809 20:28:46.470224 32367 raft_consensus.cc:348] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2a17c7e61f4faaa8f503f2670153b8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42819 } } peers { permanent_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 43081 } } peers { permanent_uuid: "5b77093b4b8a4fd489210bffb62f97b8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43821 } }
I20260809 20:28:46.470639 32345 heartbeater.cc:503] Master 127.12.78.62:35913 was elected leader, sending a full tablet report...
I20260809 20:28:46.470680 32367 raft_consensus.cc:374] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:46.470480 32366 consensus_queue.cc:260] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2a17c7e61f4faaa8f503f2670153b8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42819 } } peers { permanent_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 43081 } } peers { permanent_uuid: "5b77093b4b8a4fd489210bffb62f97b8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43821 } }
I20260809 20:28:46.470772 32367 raft_consensus.cc:729] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2a17c7e61f4faaa8f503f2670153b8, State: Initialized, Role: FOLLOWER
I20260809 20:28:46.471498 32368 ts_tablet_manager.cc:1428] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260809 20:28:46.472090 32367 consensus_queue.cc:260] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2a17c7e61f4faaa8f503f2670153b8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42819 } } peers { permanent_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 43081 } } peers { permanent_uuid: "5b77093b4b8a4fd489210bffb62f97b8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43821 } }
I20260809 20:28:46.473384 32366 ts_tablet_manager.cc:1428] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260809 20:28:46.474303 32201 heartbeater.cc:503] Master 127.12.78.62:35913 was elected leader, sending a full tablet report...
I20260809 20:28:46.475157 32367 ts_tablet_manager.cc:1428] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
W20260809 20:28:46.516389 32061 tablet.cc:2369] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:28:46.575184 32202 tablet.cc:2369] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:28:46.594372 32346 tablet.cc:2369] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:28:46.676159 32373 raft_consensus.cc:482] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:28:46.676489 32373 raft_consensus.cc:504] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2a17c7e61f4faaa8f503f2670153b8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42819 } } peers { permanent_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 43081 } } peers { permanent_uuid: "5b77093b4b8a4fd489210bffb62f97b8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43821 } }
I20260809 20:28:46.678361 32373 leader_election.cc:288] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d2a17c7e61f4faaa8f503f2670153b8 (127.12.78.2:42819), 5b77093b4b8a4fd489210bffb62f97b8 (127.12.78.3:43821)
I20260809 20:28:46.685945 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.679201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:53661 (local address 127.12.78.3:43821)
0809 20:28:46.679531 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:46.679538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:46.680277 (+ 739us) server_negotiation.cc:407] Connection header received
0809 20:28:46.680487 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:46.680492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:46.680602 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:46.680701 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:46.681326 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.682102 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.684034 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.684943 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.685088 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.685197 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:46.685533 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 20:28:46.685664 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 20:28:46.685971 32361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.678952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:44039 (local address 127.12.78.2:42819)
0809 20:28:46.679180 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:46.679189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:46.680086 (+ 897us) server_negotiation.cc:407] Connection header received
0809 20:28:46.680286 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:46.680292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:46.680373 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:46.680461 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:46.681168 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.681950 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.683989 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.684909 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.685051 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.685165 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:46.685559 (+ 394us) server_negotiation.cc:299] Negotiation successful
0809 20:28:46.685689 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 20:28:46.686110 32375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.678997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:43821 (local address 127.12.78.1:53661)
0809 20:28:46.680140 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:46.680157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:46.680331 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:46.680722 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:46.680726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:46.680754 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:46.681155 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:46.681166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.682237 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.682241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:46.683871 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:46.683883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.685209 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.685215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:46.685364 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.685387 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 20:28:46.685495 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 20:28:46.685689 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":62,"thread_start_us":462,"threads_started":1}
I20260809 20:28:46.686506 32377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.678879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:42819 (local address 127.12.78.1:44039)
0809 20:28:46.679966 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:46.679990 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:46.680136 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:46.680512 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:46.680516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:46.680603 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:46.681018 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:46.681031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.682085 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.682090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:46.683829 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:46.683844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.685156 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.685162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:46.685359 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.685384 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 20:28:46.685490 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 20:28:46.685664 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":52,"thread_start_us":910,"threads_started":1}
I20260809 20:28:46.687244 32299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268b6db41f6d4d9f8f075e6c990d6c50" candidate_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b77093b4b8a4fd489210bffb62f97b8" is_pre_election: true
I20260809 20:28:46.687569 32153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268b6db41f6d4d9f8f075e6c990d6c50" candidate_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2a17c7e61f4faaa8f503f2670153b8" is_pre_election: true
I20260809 20:28:46.687752 32299 raft_consensus.cc:2454] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ccf746e3dd4724a6cae38fd0c2fd20 in term 0.
I20260809 20:28:46.688062 32153 raft_consensus.cc:2454] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ccf746e3dd4724a6cae38fd0c2fd20 in term 0.
I20260809 20:28:46.688468 31942 leader_election.cc:302] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b77093b4b8a4fd489210bffb62f97b8, 74ccf746e3dd4724a6cae38fd0c2fd20; no voters:
I20260809 20:28:46.688942 32373 raft_consensus.cc:2781] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:28:46.689109 32373 raft_consensus.cc:482] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:28:46.689260 32373 raft_consensus.cc:3037] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:46.691627 32373 raft_consensus.cc:504] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2a17c7e61f4faaa8f503f2670153b8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42819 } } peers { permanent_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 43081 } } peers { permanent_uuid: "5b77093b4b8a4fd489210bffb62f97b8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43821 } }
I20260809 20:28:46.692255 32373 leader_election.cc:288] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [CANDIDATE]: Term 1 election: Requested vote from peers 5d2a17c7e61f4faaa8f503f2670153b8 (127.12.78.2:42819), 5b77093b4b8a4fd489210bffb62f97b8 (127.12.78.3:43821)
I20260809 20:28:46.692708 32153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268b6db41f6d4d9f8f075e6c990d6c50" candidate_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2a17c7e61f4faaa8f503f2670153b8"
I20260809 20:28:46.692788 32299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268b6db41f6d4d9f8f075e6c990d6c50" candidate_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b77093b4b8a4fd489210bffb62f97b8"
I20260809 20:28:46.692916 32153 raft_consensus.cc:3037] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:46.693011 32299 raft_consensus.cc:3037] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:46.695228 32299 raft_consensus.cc:2454] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ccf746e3dd4724a6cae38fd0c2fd20 in term 1.
I20260809 20:28:46.695231 32153 raft_consensus.cc:2454] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ccf746e3dd4724a6cae38fd0c2fd20 in term 1.
I20260809 20:28:46.695642 31942 leader_election.cc:302] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b77093b4b8a4fd489210bffb62f97b8, 74ccf746e3dd4724a6cae38fd0c2fd20; no voters:
I20260809 20:28:46.695961 32373 raft_consensus.cc:2781] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:28:46.696405 32373 raft_consensus.cc:686] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [term 1 LEADER]: Becoming Leader. State: Replica: 74ccf746e3dd4724a6cae38fd0c2fd20, State: Running, Role: LEADER
I20260809 20:28:46.696982 32373 consensus_queue.cc:237] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2a17c7e61f4faaa8f503f2670153b8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42819 } } peers { permanent_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 43081 } } peers { permanent_uuid: "5b77093b4b8a4fd489210bffb62f97b8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43821 } }
I20260809 20:28:46.701599 31857 catalog_manager.cc:5495] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74ccf746e3dd4724a6cae38fd0c2fd20 (127.12.78.1). New cstate: current_term: 1 leader_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2a17c7e61f4faaa8f503f2670153b8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74ccf746e3dd4724a6cae38fd0c2fd20" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 43081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b77093b4b8a4fd489210bffb62f97b8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43821 } health_report { overall_health: UNKNOWN } } }
I20260809 20:28:46.734092 32365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.728691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.12.78.1:43081)
0809 20:28:46.728903 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:46.728909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:46.728976 (+ 67us) server_negotiation.cc:407] Connection header received
0809 20:28:46.729215 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:46.729219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:46.729316 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:46.729498 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:28:46.729863 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.730595 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.731971 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.732362 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.732456 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.732987 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:28:46.733491 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:28:46.733628 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:46.733736 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 20:28:46.733775 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 20:28:46.741940 32153 raft_consensus.cc:1261] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8 [term 1 FOLLOWER]: Refusing update from remote peer 74ccf746e3dd4724a6cae38fd0c2fd20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:28:46.741940 32299 raft_consensus.cc:1261] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8 [term 1 FOLLOWER]: Refusing update from remote peer 74ccf746e3dd4724a6cae38fd0c2fd20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:28:46.742868 32378 consensus_queue.cc:1035] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d2a17c7e61f4faaa8f503f2670153b8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 42819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:28:46.743220 32373 consensus_queue.cc:1035] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b77093b4b8a4fd489210bffb62f97b8" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 43821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:28:46.769065 31858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:28:46.769421 31858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:28:46.773218 31858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 520039dadf9a4112a858b55a06f6ba82: Sending DeleteTablet for 3 replicas of tablet 268b6db41f6d4d9f8f075e6c990d6c50
I20260809 20:28:46.774396 31991 tablet_service.cc:1514] Processing DeleteTablet for tablet 268b6db41f6d4d9f8f075e6c990d6c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:46 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:33560
I20260809 20:28:46.774402 32273 tablet_service.cc:1514] Processing DeleteTablet for tablet 268b6db41f6d4d9f8f075e6c990d6c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:46 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:41442
I20260809 20:28:46.774569 32131 tablet_service.cc:1514] Processing DeleteTablet for tablet 268b6db41f6d4d9f8f075e6c990d6c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:46 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:57508
I20260809 20:28:46.776249 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 31925
I20260809 20:28:46.793919 32393 tablet_replica.cc:330] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8: stopping tablet replica
I20260809 20:28:46.794152 32392 tablet_replica.cc:330] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20: stopping tablet replica
I20260809 20:28:46.794409 32394 tablet_replica.cc:330] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8: stopping tablet replica
I20260809 20:28:46.794430 32393 raft_consensus.cc:2229] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:28:46.794521 32392 raft_consensus.cc:2229] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:28:46.794888 32394 raft_consensus.cc:2229] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:28:46.795015 32393 raft_consensus.cc:2258] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:46.795153 32392 raft_consensus.cc:2258] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:46.795387 32394 raft_consensus.cc:2258] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:46.797423 32393 ts_tablet_manager.cc:1905] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:46.797415 32392 ts_tablet_manager.cc:1905] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:46.800706 32394 ts_tablet_manager.cc:1905] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:46.804837 32393 ts_tablet_manager.cc:1918] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:46.805004 32393 log.cc:1196] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/268b6db41f6d4d9f8f075e6c990d6c50
I20260809 20:28:46.805418 32392 ts_tablet_manager.cc:1918] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:46.805545 32393 ts_tablet_manager.cc:1939] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5b77093b4b8a4fd489210bffb62f97b8: Deleting consensus metadata
I20260809 20:28:46.805629 32392 log.cc:1196] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/268b6db41f6d4d9f8f075e6c990d6c50
I20260809 20:28:46.806169 32392 ts_tablet_manager.cc:1939] T 268b6db41f6d4d9f8f075e6c990d6c50 P 74ccf746e3dd4724a6cae38fd0c2fd20: Deleting consensus metadata
I20260809 20:28:46.807585 31845 catalog_manager.cc:4841] TS 5b77093b4b8a4fd489210bffb62f97b8 (127.12.78.3:43821): tablet 268b6db41f6d4d9f8f075e6c990d6c50 (table test-table [id=222e7dd1489d48aa8346076751a8882a]) successfully deleted
I20260809 20:28:46.807659 32394 ts_tablet_manager.cc:1918] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:46.807824 32394 log.cc:1196] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/268b6db41f6d4d9f8f075e6c990d6c50
I20260809 20:28:46.808027 31847 catalog_manager.cc:4841] TS 74ccf746e3dd4724a6cae38fd0c2fd20 (127.12.78.1:43081): tablet 268b6db41f6d4d9f8f075e6c990d6c50 (table test-table [id=222e7dd1489d48aa8346076751a8882a]) successfully deleted
I20260809 20:28:46.808387 32394 ts_tablet_manager.cc:1939] T 268b6db41f6d4d9f8f075e6c990d6c50 P 5d2a17c7e61f4faaa8f503f2670153b8: Deleting consensus metadata
I20260809 20:28:46.810127 31847 catalog_manager.cc:4841] TS 5d2a17c7e61f4faaa8f503f2670153b8 (127.12.78.2:42819): tablet 268b6db41f6d4d9f8f075e6c990d6c50 (table test-table [id=222e7dd1489d48aa8346076751a8882a]) successfully deleted
Aug 09 20:28:46 dist-test-slave-rnrw krb5kdc[31814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:28:46.819267 32365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.777072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.12.78.1:43081)
0809 20:28:46.777311 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:46.777317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:46.784339 (+ 7022us) server_negotiation.cc:407] Connection header received
0809 20:28:46.785269 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:46.785276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:46.785389 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:46.785485 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:46.789944 (+ 4459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.791624 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.797141 (+ 5517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.797890 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.798011 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.812198 (+ 14187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:46.812214 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:46.812232 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:46.812294 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:46.816281 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:46.817071 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:46.817081 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:46.817085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:46.817271 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:46.817643 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:46.817648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:46.817652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:46.818111 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:46.818414 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:46.818651 (+ 237us) server_negotiation.cc:299] Negotiation successful
0809 20:28:46.818902 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260809 20:28:46.820096 32054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:46.919559 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 31925
I20260809 20:28:46.940634 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 32066
Aug 09 20:28:46 dist-test-slave-rnrw krb5kdc[31814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:28:46.956274 32361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:46.941476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.12.78.2:42819)
0809 20:28:46.941657 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:46.941663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:46.941756 (+ 93us) server_negotiation.cc:407] Connection header received
0809 20:28:46.941963 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:46.941969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:46.942036 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:46.942149 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:46.943422 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.944217 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.945601 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:46.946014 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:46.946150 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:46.950491 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:46.950507 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:46.950523 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:46.950562 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:46.953415 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:46.954142 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:46.954149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:46.954152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:46.954282 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:46.954665 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:46.954671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:46.954674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:46.955061 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:46.955387 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:46.955684 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 20:28:46.955905 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 20:28:46.957108 32196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:47.049955 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 32066
I20260809 20:28:47.068351 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 32208
Aug 09 20:28:47 dist-test-slave-rnrw krb5kdc[31814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:28:47.083350 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:47.069151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.12.78.3:43821)
0809 20:28:47.069317 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:47.069321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:47.069343 (+ 22us) server_negotiation.cc:407] Connection header received
0809 20:28:47.069509 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:47.069513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:47.069577 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:47.069663 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:47.071063 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:47.071803 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:47.073086 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:47.073537 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:47.073637 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:47.077911 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:47.077926 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:47.077943 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:47.077981 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:47.080515 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:47.081268 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:47.081276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:47.081279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:47.081424 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:47.081791 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:47.081796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:47.081799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:47.082160 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:47.082468 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:47.082781 (+ 313us) server_negotiation.cc:299] Negotiation successful
0809 20:28:47.083003 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 20:28:47.084082 32339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:47.173480 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 32208
I20260809 20:28:47.193378 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 31827
I20260809 20:28:47.194636 31892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:47.309280 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 31827
2026-08-09T20:28:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4685 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T20:28:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:28:47Z Disabled control of system clock
I20260809 20:28:47.395658 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:34315
--webserver_interface=127.12.78.62
--webserver_port=0
--builtin_ntp_servers=127.12.78.20:39845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:34315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 20:28:47.740705 32404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:47.740991 32404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:47.741073 32404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:47.748124 32404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:28:47.748230 32404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:47.748275 32404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:47.748333 32404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:28:47.748379 32404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:28:47.757565 32404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:39845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:34315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:34315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:47.758811 32404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:47.760644 32404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:47.772456 32410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:47.772493 32409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:47.772470 32412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:47.773252 32404 server_base.cc:1034] running on GCE node
I20260809 20:28:47.774048 32404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:47.775446 32404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:47.776652 32404 hybrid_clock.cc:648] HybridClock initialized: now 1786307327776593 us; error 44 us; skew 500 ppm
I20260809 20:28:47.777006 32404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:28:47.779242 32404 webserver.cc:460] Webserver started at http://127.12.78.62:35813/ using document root <none> and password file <none>
I20260809 20:28:47.779922 32404 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:47.780048 32404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:47.780388 32404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:47.783315 32404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "73dfaea8eb114756b83f262830fc8f23"
format_stamp: "Formatted at 2026-08-09 20:28:47 on dist-test-slave-rnrw"
server_key: "73ab20a703dbbad80c8b2ce8bc5b9c5c"
server_key_iv: "cad50b1f2c5154e75d04d7580c741288"
server_key_version: "encryptionkey@0"
I20260809 20:28:47.784117 32404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "73dfaea8eb114756b83f262830fc8f23"
format_stamp: "Formatted at 2026-08-09 20:28:47 on dist-test-slave-rnrw"
server_key: "73ab20a703dbbad80c8b2ce8bc5b9c5c"
server_key_iv: "cad50b1f2c5154e75d04d7580c741288"
server_key_version: "encryptionkey@0"
I20260809 20:28:47.789146 32404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 20:28:47.792449 32420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:47.793812 32404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:28:47.793994 32404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "73dfaea8eb114756b83f262830fc8f23"
format_stamp: "Formatted at 2026-08-09 20:28:47 on dist-test-slave-rnrw"
server_key: "73ab20a703dbbad80c8b2ce8bc5b9c5c"
server_key_iv: "cad50b1f2c5154e75d04d7580c741288"
server_key_version: "encryptionkey@0"
I20260809 20:28:47.794204 32404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:47.814013 32404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:47.814855 32404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:47.815160 32404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:47.833907 32404 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:34315
I20260809 20:28:47.833947 32472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:34315 every 8 connection(s)
I20260809 20:28:47.835286 32404 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:28:47.839152 32473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:47.841740 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 32404
I20260809 20:28:47.841938 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:28:47.842531 12600 external_mini_cluster.cc:1428] Setting key 59810a8d29f190f226a106c29671b676
I20260809 20:28:47.852289 32473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23: Bootstrap starting.
I20260809 20:28:47.855944 32473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:47.856480 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:47.844773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.12.78.62:34315)
0809 20:28:47.845885 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:47.845909 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:47.845962 (+ 53us) server_negotiation.cc:407] Connection header received
0809 20:28:47.847722 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:47.847743 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:47.848186 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:47.848607 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:47.849187 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:47.850516 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:47.851864 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:47.852660 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:47.852848 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:47.854082 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:47.854091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:47.854104 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:47.854702 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:47.854794 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:47.854820 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:47.854947 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:47.855165 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 20:28:47.855469 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":353,"threads_started":1}
I20260809 20:28:47.857232 32473 log.cc:824] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:47.860515 32473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23: No bootstrap required, opened a new log
I20260809 20:28:47.869037 32473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dfaea8eb114756b83f262830fc8f23" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 34315 } }
I20260809 20:28:47.869480 32473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:47.869602 32473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73dfaea8eb114756b83f262830fc8f23, State: Initialized, Role: FOLLOWER
I20260809 20:28:47.870205 32473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dfaea8eb114756b83f262830fc8f23" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 34315 } }
I20260809 20:28:47.870464 32473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:28:47.870623 32473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:28:47.870808 32473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:47.873613 32473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dfaea8eb114756b83f262830fc8f23" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 34315 } }
I20260809 20:28:47.874145 32473 leader_election.cc:302] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73dfaea8eb114756b83f262830fc8f23; no voters:
I20260809 20:28:47.874819 32473 leader_election.cc:288] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:28:47.874951 32478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:28:47.875854 32478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [term 1 LEADER]: Becoming Leader. State: Replica: 73dfaea8eb114756b83f262830fc8f23, State: Running, Role: LEADER
I20260809 20:28:47.876384 32478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dfaea8eb114756b83f262830fc8f23" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 34315 } }
I20260809 20:28:47.877341 32473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:28:47.879222 32479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73dfaea8eb114756b83f262830fc8f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dfaea8eb114756b83f262830fc8f23" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 34315 } } }
I20260809 20:28:47.879185 32480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73dfaea8eb114756b83f262830fc8f23. Latest consensus state: current_term: 1 leader_uuid: "73dfaea8eb114756b83f262830fc8f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dfaea8eb114756b83f262830fc8f23" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 34315 } } }
I20260809 20:28:47.880218 32480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:47.879673 32479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:47.884337 32484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:28:47.895495 32484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:28:47.905077 32484 catalog_manager.cc:1347] Generated new cluster ID: 6e623b5d6a874dd6a59a19f9b2aeaabc
I20260809 20:28:47.905215 32484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:28:47.933005 32484 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:28:47.934653 32484 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:28:47.943318 32484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23: Generated new TSK 0
I20260809 20:28:47.944254 32484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:28:47.953616 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:34315
--builtin_ntp_servers=127.12.78.20:39845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 20:28:48.294550 32500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:48.294860 32500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:48.295014 32500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:48.302618 32500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:48.302721 32500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:48.302887 32500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:28:48.313424 32500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:39845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:34315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:48.314734 32500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:48.316512 32500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:48.330463 32508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:48.330435 32505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:48.330366 32506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:48.331146 32500 server_base.cc:1034] running on GCE node
I20260809 20:28:48.331722 32500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:48.332731 32500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:48.334017 32500 hybrid_clock.cc:648] HybridClock initialized: now 1786307328333924 us; error 75 us; skew 500 ppm
I20260809 20:28:48.334358 32500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:28:48.336774 32500 webserver.cc:460] Webserver started at http://127.12.78.1:46213/ using document root <none> and password file <none>
I20260809 20:28:48.337481 32500 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:48.337620 32500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:48.337939 32500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:48.340919 32500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "148e5a4092f64857a683785cff7be05d"
format_stamp: "Formatted at 2026-08-09 20:28:48 on dist-test-slave-rnrw"
server_key: "ad627c09720800875d149f998abb5079"
server_key_iv: "3e28c5d6a9af807480086254f79d3044"
server_key_version: "encryptionkey@0"
I20260809 20:28:48.341712 32500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "148e5a4092f64857a683785cff7be05d"
format_stamp: "Formatted at 2026-08-09 20:28:48 on dist-test-slave-rnrw"
server_key: "ad627c09720800875d149f998abb5079"
server_key_iv: "3e28c5d6a9af807480086254f79d3044"
server_key_version: "encryptionkey@0"
I20260809 20:28:48.347128 32500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260809 20:28:48.350603 32514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:48.351878 32500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 20:28:48.352051 32500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "148e5a4092f64857a683785cff7be05d"
format_stamp: "Formatted at 2026-08-09 20:28:48 on dist-test-slave-rnrw"
server_key: "ad627c09720800875d149f998abb5079"
server_key_iv: "3e28c5d6a9af807480086254f79d3044"
server_key_version: "encryptionkey@0"
I20260809 20:28:48.352267 32500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:48.369685 32500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:48.370497 32500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:48.370779 32500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:48.371822 32500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:48.373239 32500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:48.373373 32500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:48.373493 32500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:48.373586 32500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:48.407986 32500 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:33417
I20260809 20:28:48.408042 32629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:33417 every 8 connection(s)
I20260809 20:28:48.409418 32500 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:28:48.412176 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 32500
I20260809 20:28:48.412372 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:28:48.412914 12600 external_mini_cluster.cc:1428] Setting key 8748562358222aad773eb5b3a0917a53
I20260809 20:28:48.415483 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:34315
--builtin_ntp_servers=127.12.78.20:39845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 20:28:48.423305 32632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:48.412062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:60177 (local address 127.12.78.62:34315)
0809 20:28:48.412714 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:48.412720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:48.414023 (+ 1303us) server_negotiation.cc:407] Connection header received
0809 20:28:48.415792 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:48.415800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:48.415886 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:48.416010 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:48.417074 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:48.418261 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:48.419898 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:48.420646 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:48.420800 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:48.422184 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:48.422193 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:48.422199 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:48.422225 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:48.422284 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:48.422304 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:48.422406 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:48.422733 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 20:28:48.422851 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":332,"threads_started":1}
I20260809 20:28:48.424021 32634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:48.412769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:34315 (local address 127.12.78.1:60177)
0809 20:28:48.413847 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:48.413900 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:48.415506 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:48.416193 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:48.416201 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:48.416281 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:48.416910 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:48.416927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:48.418417 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:48.418420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:48.419736 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:48.419748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:48.420859 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:48.420865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:48.421121 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:48.421866 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:28:48.421869 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:48.421906 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:28:48.421991 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:48.422500 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:48.422510 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:48.422517 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 20:28:48.422685 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 20:28:48.422959 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":375,"threads_started":1}
I20260809 20:28:48.425675 32630 heartbeater.cc:347] Connected to a master server at 127.12.78.62:34315
I20260809 20:28:48.425987 32630 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:48.426724 32630 heartbeater.cc:511] Master 127.12.78.62:34315 requested a full tablet report, sending...
I20260809 20:28:48.428920 32438 ts_manager.cc:194] Registered new tserver with Master: 148e5a4092f64857a683785cff7be05d (127.12.78.1:33417)
I20260809 20:28:48.430434 32438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.78.1:60177
W20260809 20:28:48.763039 32635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:48.763306 32635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:48.763446 32635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:48.770633 32635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:48.770731 32635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:48.770907 32635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:28:48.783191 32635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:39845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:34315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:48.785065 32635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:48.787110 32635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:48.800196 32641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:48.800168 32644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:48.800264 32642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:48.801187 32635 server_base.cc:1034] running on GCE node
I20260809 20:28:48.801764 32635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:48.802681 32635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:48.803889 32635 hybrid_clock.cc:648] HybridClock initialized: now 1786307328803839 us; error 37 us; skew 500 ppm
I20260809 20:28:48.804221 32635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:28:48.806581 32635 webserver.cc:460] Webserver started at http://127.12.78.2:41503/ using document root <none> and password file <none>
I20260809 20:28:48.807282 32635 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:48.807394 32635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:48.807739 32635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:48.810727 32635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "a337e8fce8f6404a9f0cfef08bf5b03d"
format_stamp: "Formatted at 2026-08-09 20:28:48 on dist-test-slave-rnrw"
server_key: "a610dddae4523feca9f9f6c550e05fbc"
server_key_iv: "91e936521d9ab9b4c65f2597e67610f8"
server_key_version: "encryptionkey@0"
I20260809 20:28:48.811527 32635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "a337e8fce8f6404a9f0cfef08bf5b03d"
format_stamp: "Formatted at 2026-08-09 20:28:48 on dist-test-slave-rnrw"
server_key: "a610dddae4523feca9f9f6c550e05fbc"
server_key_iv: "91e936521d9ab9b4c65f2597e67610f8"
server_key_version: "encryptionkey@0"
I20260809 20:28:48.816721 32635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 20:28:48.820016 32650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:48.821355 32635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:28:48.821529 32635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "a337e8fce8f6404a9f0cfef08bf5b03d"
format_stamp: "Formatted at 2026-08-09 20:28:48 on dist-test-slave-rnrw"
server_key: "a610dddae4523feca9f9f6c550e05fbc"
server_key_iv: "91e936521d9ab9b4c65f2597e67610f8"
server_key_version: "encryptionkey@0"
I20260809 20:28:48.821736 32635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:48.833220 32635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:48.834096 32635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:48.834365 32635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:48.835353 32635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:48.836946 32635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:48.837064 32635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:48.837198 32635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:48.837281 32635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:48.872375 32635 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:37251
I20260809 20:28:48.872423 32766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:37251 every 8 connection(s)
I20260809 20:28:48.873728 32635 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:28:48.877599 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 32635
I20260809 20:28:48.877804 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:28:48.878355 12600 external_mini_cluster.cc:1428] Setting key 8c3af7f0ce7815c683d3dcef7aca7596
I20260809 20:28:48.881453 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:34315
--builtin_ntp_servers=127.12.78.20:39845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 20:28:48.888209 32632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:48.876317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:47945 (local address 127.12.78.62:34315)
0809 20:28:48.876528 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:48.876534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:48.878191 (+ 1657us) server_negotiation.cc:407] Connection header received
0809 20:28:48.880539 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:48.880545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:48.880655 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:48.880770 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:48.882017 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:48.883220 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:48.884925 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:48.885657 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:48.885793 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:48.887215 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:48.887224 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:48.887231 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:48.887258 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:48.887301 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:48.887318 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:48.887413 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:48.887689 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 20:28:48.887805 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 20:28:48.889153 301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:48.876910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:34315 (local address 127.12.78.2:47945)
0809 20:28:48.877996 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:48.878043 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:48.880213 (+ 2170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:48.881036 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:48.881045 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:48.881140 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:48.881841 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:48.881861 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:48.883386 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:48.883390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:48.884767 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:48.884782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:48.885890 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:48.885903 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:48.886123 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:48.886885 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:28:48.886888 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:48.886929 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:28:48.887022 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:48.887468 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:48.887478 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:48.887484 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 20:28:48.887651 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 20:28:48.887949 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":365,"threads_started":1}
I20260809 20:28:48.890605 32767 heartbeater.cc:347] Connected to a master server at 127.12.78.62:34315
I20260809 20:28:48.890929 32767 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:48.891649 32767 heartbeater.cc:511] Master 127.12.78.62:34315 requested a full tablet report, sending...
I20260809 20:28:48.893230 32438 ts_manager.cc:194] Registered new tserver with Master: a337e8fce8f6404a9f0cfef08bf5b03d (127.12.78.2:37251)
I20260809 20:28:48.894279 32438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.78.2:47945
W20260809 20:28:49.223320 302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:49.223654 302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:49.223812 302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:49.231315 302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:49.231482 302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:49.231673 302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:28:49.241170 302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:39845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:34315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:49.242555 302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:49.244455 302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:49.258134 311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:49.258247 308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:49.258229 309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:49.259109 302 server_base.cc:1034] running on GCE node
I20260809 20:28:49.259773 302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:49.260824 302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:49.262061 302 hybrid_clock.cc:648] HybridClock initialized: now 1786307329262017 us; error 35 us; skew 500 ppm
I20260809 20:28:49.262399 302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:28:49.264734 302 webserver.cc:460] Webserver started at http://127.12.78.3:33221/ using document root <none> and password file <none>
I20260809 20:28:49.265441 302 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:49.265580 302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:49.265915 302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:49.268954 302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "78cb732272164c69924afb8acad2df2d"
format_stamp: "Formatted at 2026-08-09 20:28:49 on dist-test-slave-rnrw"
server_key: "4b0ac6a294269353e631a8061a0041d9"
server_key_iv: "d314711ef1551182a61b7fef321c101b"
server_key_version: "encryptionkey@0"
I20260809 20:28:49.269765 302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "78cb732272164c69924afb8acad2df2d"
format_stamp: "Formatted at 2026-08-09 20:28:49 on dist-test-slave-rnrw"
server_key: "4b0ac6a294269353e631a8061a0041d9"
server_key_iv: "d314711ef1551182a61b7fef321c101b"
server_key_version: "encryptionkey@0"
I20260809 20:28:49.274852 302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:28:49.278501 317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:49.279805 302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 20:28:49.279979 302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "78cb732272164c69924afb8acad2df2d"
format_stamp: "Formatted at 2026-08-09 20:28:49 on dist-test-slave-rnrw"
server_key: "4b0ac6a294269353e631a8061a0041d9"
server_key_iv: "d314711ef1551182a61b7fef321c101b"
server_key_version: "encryptionkey@0"
I20260809 20:28:49.280208 302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:49.300329 302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:49.301240 302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:49.301548 302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:49.302587 302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:49.304041 302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:49.304169 302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:49.304311 302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:49.304399 302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:49.339303 302 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:37749
I20260809 20:28:49.339395 435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:37749 every 8 connection(s)
I20260809 20:28:49.340740 302 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:28:49.342581 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 302
I20260809 20:28:49.342792 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:28:49.343369 12600 external_mini_cluster.cc:1428] Setting key 6120ec88be0cb979cc1b822c302a6bf3
I20260809 20:28:49.352849 32632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.343373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:60775 (local address 127.12.78.62:34315)
0809 20:28:49.343589 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:49.343597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:49.345170 (+ 1573us) server_negotiation.cc:407] Connection header received
0809 20:28:49.346896 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:49.346900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:49.346962 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:49.347044 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:49.348040 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.348863 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.350350 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.350758 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.350850 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.352115 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:49.352119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:49.352122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:49.352140 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:49.352166 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:49.352176 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:49.352248 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:49.352451 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 20:28:49.352529 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 20:28:49.353789 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.343935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:34315 (local address 127.12.78.3:60775)
0809 20:28:49.345000 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:49.345049 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:49.346657 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:49.347244 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:49.347252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:49.347332 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:49.347897 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:49.347913 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.349001 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.349005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:49.350230 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:49.350241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.350936 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.350940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:49.351144 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.351840 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:28:49.351843 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:49.351880 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:28:49.351957 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:49.352259 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:49.352266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:49.352272 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 20:28:49.352410 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 20:28:49.352747 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":349,"threads_started":1}
I20260809 20:28:49.355144 436 heartbeater.cc:347] Connected to a master server at 127.12.78.62:34315
I20260809 20:28:49.355469 436 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:49.356252 436 heartbeater.cc:511] Master 127.12.78.62:34315 requested a full tablet report, sending...
I20260809 20:28:49.357827 32438 ts_manager.cc:194] Registered new tserver with Master: 78cb732272164c69924afb8acad2df2d (127.12.78.3:37749)
I20260809 20:28:49.358618 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:28:49.358767 32438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.78.3:60775
I20260809 20:28:49.368494 32632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.362885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.12.78.62:34315)
0809 20:28:49.363085 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:49.363089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:49.363551 (+ 462us) server_negotiation.cc:407] Connection header received
0809 20:28:49.363686 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:49.363690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:49.363757 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:49.363837 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:49.364286 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.365027 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.366323 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.366745 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.366835 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.367913 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:49.367917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:49.367920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:49.367936 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:49.367962 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:49.367973 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:49.368092 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:49.368238 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 20:28:49.368280 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 20:28:49.372223 32438 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:28:49.379216 32438 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:28:49.402276 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.391186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:37749 (local address 127.0.0.1:37408)
0809 20:28:49.392031 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:49.392074 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:49.392239 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:49.393737 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:49.393744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:49.393776 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:49.394267 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:49.394283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.396605 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.396610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:49.399463 (+ 2853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:49.399482 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.401276 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.401282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:49.401478 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.401556 (+ 78us) client_negotiation.cc:769] Sending connection context
0809 20:28:49.401673 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 20:28:49.401868 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":93,"thread_start_us":492,"threads_started":1}
I20260809 20:28:49.402534 451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.391879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.12.78.2:37251)
0809 20:28:49.392690 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:49.392701 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:49.393237 (+ 536us) server_negotiation.cc:407] Connection header received
0809 20:28:49.393512 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:49.393522 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:49.393741 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:49.393904 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:49.394873 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.396430 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.399995 (+ 3565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.401421 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.401561 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.401781 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:49.402006 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 20:28:49.402196 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":495,"threads_started":1}
I20260809 20:28:49.402932 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.392002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.12.78.3:37749)
0809 20:28:49.393071 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:49.393079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:49.393114 (+ 35us) server_negotiation.cc:407] Connection header received
0809 20:28:49.393216 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:49.393224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:49.393486 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:49.394476 (+ 990us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:49.394774 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.396385 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.399684 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.401053 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.402027 (+ 974us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.402308 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:49.402381 (+ 73us) server_negotiation.cc:299] Negotiation successful
0809 20:28:49.402547 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":716,"threads_started":1}
I20260809 20:28:49.403302 450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.391365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:37251 (local address 127.0.0.1:48540)
0809 20:28:49.393131 (+ 1766us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:49.393151 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:49.393337 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:49.394207 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:49.394217 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:49.394246 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:49.394660 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:49.394672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.396617 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.396620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:49.399798 (+ 3178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:49.399811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.401619 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.401624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:49.401807 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.401836 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 20:28:49.402787 (+ 951us) client_negotiation.cc:240] Negotiation successful
0809 20:28:49.402942 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1600,"mutex_wait_us":61,"thread_start_us":1496,"threads_started":1}
I20260809 20:28:49.406586 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.393208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:33417 (local address 127.0.0.1:56314)
0809 20:28:49.397098 (+ 3890us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:49.397117 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:49.397229 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:49.399025 (+ 1796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:49.399029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:49.399053 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:49.399505 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:49.399520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.401941 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.401949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:49.403933 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:49.403953 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.405855 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.405864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:49.406033 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.406066 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 20:28:49.406179 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 20:28:49.406305 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3741,"thread_start_us":3588,"threads_started":1}
I20260809 20:28:49.406745 454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.393521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.12.78.1:33417)
0809 20:28:49.398587 (+ 5066us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:49.398593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:49.398617 (+ 24us) server_negotiation.cc:407] Connection header received
0809 20:28:49.398690 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:49.398694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:49.398843 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:49.399037 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:49.399690 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.401790 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.404184 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.405639 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.405785 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.406011 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:49.406238 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 20:28:49.406400 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4907,"thread_start_us":4782,"threads_started":1}
I20260809 20:28:49.407181 371 tablet_service.cc:1467] Processing CreateTablet for tablet 5e98acde40db4464872b2b850fa820ff (DEFAULT_TABLE table=test-table [id=9ead03f10b384d6ca88ba73738f9c09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:49.408893 371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e98acde40db4464872b2b850fa820ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:49.409895 32564 tablet_service.cc:1467] Processing CreateTablet for tablet 5e98acde40db4464872b2b850fa820ff (DEFAULT_TABLE table=test-table [id=9ead03f10b384d6ca88ba73738f9c09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:49.411195 32564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e98acde40db4464872b2b850fa820ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:49.407161 32700 tablet_service.cc:1467] Processing CreateTablet for tablet 5e98acde40db4464872b2b850fa820ff (DEFAULT_TABLE table=test-table [id=9ead03f10b384d6ca88ba73738f9c09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:49.418051 32700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e98acde40db4464872b2b850fa820ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:49.422303 455 tablet_bootstrap.cc:492] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d: Bootstrap starting.
I20260809 20:28:49.425043 456 tablet_bootstrap.cc:492] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d: Bootstrap starting.
I20260809 20:28:49.426105 455 tablet_bootstrap.cc:654] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:49.427457 455 log.cc:824] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:49.428740 456 tablet_bootstrap.cc:654] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:49.430022 456 log.cc:824] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:49.430938 455 tablet_bootstrap.cc:492] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d: No bootstrap required, opened a new log
I20260809 20:28:49.431294 455 ts_tablet_manager.cc:1397] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 20:28:49.433175 459 tablet_bootstrap.cc:492] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d: Bootstrap starting.
I20260809 20:28:49.434320 32630 heartbeater.cc:503] Master 127.12.78.62:34315 was elected leader, sending a full tablet report...
I20260809 20:28:49.436947 459 tablet_bootstrap.cc:654] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:49.439280 456 tablet_bootstrap.cc:492] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d: No bootstrap required, opened a new log
I20260809 20:28:49.439651 456 ts_tablet_manager.cc:1397] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260809 20:28:49.439798 459 log.cc:824] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:49.443105 459 tablet_bootstrap.cc:492] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d: No bootstrap required, opened a new log
I20260809 20:28:49.443387 459 ts_tablet_manager.cc:1397] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260809 20:28:49.443526 455 raft_consensus.cc:348] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb732272164c69924afb8acad2df2d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37749 } } peers { permanent_uuid: "a337e8fce8f6404a9f0cfef08bf5b03d" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37251 } } peers { permanent_uuid: "148e5a4092f64857a683785cff7be05d" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33417 } }
I20260809 20:28:49.443935 455 raft_consensus.cc:374] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:49.444062 455 raft_consensus.cc:729] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78cb732272164c69924afb8acad2df2d, State: Initialized, Role: FOLLOWER
I20260809 20:28:49.444746 455 consensus_queue.cc:260] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb732272164c69924afb8acad2df2d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37749 } } peers { permanent_uuid: "a337e8fce8f6404a9f0cfef08bf5b03d" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37251 } } peers { permanent_uuid: "148e5a4092f64857a683785cff7be05d" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33417 } }
I20260809 20:28:49.445886 436 heartbeater.cc:503] Master 127.12.78.62:34315 was elected leader, sending a full tablet report...
I20260809 20:28:49.446559 455 ts_tablet_manager.cc:1428] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260809 20:28:49.450409 456 raft_consensus.cc:348] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb732272164c69924afb8acad2df2d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37749 } } peers { permanent_uuid: "a337e8fce8f6404a9f0cfef08bf5b03d" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37251 } } peers { permanent_uuid: "148e5a4092f64857a683785cff7be05d" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33417 } }
I20260809 20:28:49.450919 456 raft_consensus.cc:374] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:49.451028 456 raft_consensus.cc:729] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148e5a4092f64857a683785cff7be05d, State: Initialized, Role: FOLLOWER
I20260809 20:28:49.451761 456 consensus_queue.cc:260] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb732272164c69924afb8acad2df2d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37749 } } peers { permanent_uuid: "a337e8fce8f6404a9f0cfef08bf5b03d" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37251 } } peers { permanent_uuid: "148e5a4092f64857a683785cff7be05d" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33417 } }
I20260809 20:28:49.453657 456 ts_tablet_manager.cc:1428] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260809 20:28:49.454519 459 raft_consensus.cc:348] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb732272164c69924afb8acad2df2d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37749 } } peers { permanent_uuid: "a337e8fce8f6404a9f0cfef08bf5b03d" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37251 } } peers { permanent_uuid: "148e5a4092f64857a683785cff7be05d" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33417 } }
I20260809 20:28:49.454967 459 raft_consensus.cc:374] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:49.455089 459 raft_consensus.cc:729] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a337e8fce8f6404a9f0cfef08bf5b03d, State: Initialized, Role: FOLLOWER
I20260809 20:28:49.455754 459 consensus_queue.cc:260] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb732272164c69924afb8acad2df2d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37749 } } peers { permanent_uuid: "a337e8fce8f6404a9f0cfef08bf5b03d" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37251 } } peers { permanent_uuid: "148e5a4092f64857a683785cff7be05d" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33417 } }
I20260809 20:28:49.456842 32767 heartbeater.cc:503] Master 127.12.78.62:34315 was elected leader, sending a full tablet report...
I20260809 20:28:49.457553 459 ts_tablet_manager.cc:1428] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260809 20:28:49.564638 462 raft_consensus.cc:482] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:28:49.564993 462 raft_consensus.cc:504] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb732272164c69924afb8acad2df2d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37749 } } peers { permanent_uuid: "a337e8fce8f6404a9f0cfef08bf5b03d" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37251 } } peers { permanent_uuid: "148e5a4092f64857a683785cff7be05d" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33417 } }
I20260809 20:28:49.566942 462 leader_election.cc:288] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78cb732272164c69924afb8acad2df2d (127.12.78.3:37749), a337e8fce8f6404a9f0cfef08bf5b03d (127.12.78.2:37251)
I20260809 20:28:49.574266 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.567485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:39591 (local address 127.12.78.3:37749)
0809 20:28:49.567739 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:49.567746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:49.568564 (+ 818us) server_negotiation.cc:407] Connection header received
0809 20:28:49.568809 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:49.568813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:49.568886 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:49.568993 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:49.569611 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.570549 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.572490 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.573323 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.573464 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.573573 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:49.573870 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 20:28:49.574009 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260809 20:28:49.574369 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.567511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:37749 (local address 127.12.78.1:39591)
0809 20:28:49.568390 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:49.568417 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:49.568648 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:49.569007 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:49.569012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:49.569035 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:49.569446 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:49.569455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.570683 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.570689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:49.572301 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:49.572313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.573531 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.573537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:49.573694 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.573719 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 20:28:49.573839 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 20:28:49.573989 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":68,"thread_start_us":736,"threads_started":1}
I20260809 20:28:49.574725 451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.567682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:50423 (local address 127.12.78.2:37251)
0809 20:28:49.567932 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:49.567940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:49.568659 (+ 719us) server_negotiation.cc:407] Connection header received
0809 20:28:49.568866 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:49.568872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:49.568960 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:49.569081 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:49.569815 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.570749 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.572842 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.573720 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.573888 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.574005 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:49.574300 (+ 295us) server_negotiation.cc:299] Negotiation successful
0809 20:28:49.574440 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 20:28:49.574836 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.567533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:37251 (local address 127.12.78.1:50423)
0809 20:28:49.568492 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:49.568507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:49.568714 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:49.569098 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:49.569103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:49.569145 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:49.569652 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:49.569663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.570911 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.570916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:49.572646 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:49.572659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.573942 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.573949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:49.574130 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.574156 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 20:28:49.574270 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 20:28:49.574425 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":39,"thread_start_us":477,"threads_started":1}
I20260809 20:28:49.575667 391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e98acde40db4464872b2b850fa820ff" candidate_uuid: "148e5a4092f64857a683785cff7be05d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78cb732272164c69924afb8acad2df2d" is_pre_election: true
I20260809 20:28:49.576215 391 raft_consensus.cc:2454] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148e5a4092f64857a683785cff7be05d in term 0.
I20260809 20:28:49.576201 32721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e98acde40db4464872b2b850fa820ff" candidate_uuid: "148e5a4092f64857a683785cff7be05d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a337e8fce8f6404a9f0cfef08bf5b03d" is_pre_election: true
I20260809 20:28:49.576741 32721 raft_consensus.cc:2454] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148e5a4092f64857a683785cff7be05d in term 0.
I20260809 20:28:49.577018 32515 leader_election.cc:302] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148e5a4092f64857a683785cff7be05d, 78cb732272164c69924afb8acad2df2d; no voters:
I20260809 20:28:49.577486 462 raft_consensus.cc:2781] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:28:49.577646 462 raft_consensus.cc:482] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:28:49.577762 462 raft_consensus.cc:3037] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:49.580080 462 raft_consensus.cc:504] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb732272164c69924afb8acad2df2d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37749 } } peers { permanent_uuid: "a337e8fce8f6404a9f0cfef08bf5b03d" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37251 } } peers { permanent_uuid: "148e5a4092f64857a683785cff7be05d" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33417 } }
I20260809 20:28:49.580832 462 leader_election.cc:288] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [CANDIDATE]: Term 1 election: Requested vote from peers 78cb732272164c69924afb8acad2df2d (127.12.78.3:37749), a337e8fce8f6404a9f0cfef08bf5b03d (127.12.78.2:37251)
I20260809 20:28:49.581264 391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e98acde40db4464872b2b850fa820ff" candidate_uuid: "148e5a4092f64857a683785cff7be05d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78cb732272164c69924afb8acad2df2d"
I20260809 20:28:49.581390 32721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e98acde40db4464872b2b850fa820ff" candidate_uuid: "148e5a4092f64857a683785cff7be05d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a337e8fce8f6404a9f0cfef08bf5b03d"
I20260809 20:28:49.581497 391 raft_consensus.cc:3037] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:49.581598 32721 raft_consensus.cc:3037] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:49.584873 391 raft_consensus.cc:2454] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148e5a4092f64857a683785cff7be05d in term 1.
I20260809 20:28:49.584873 32721 raft_consensus.cc:2454] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148e5a4092f64857a683785cff7be05d in term 1.
I20260809 20:28:49.585371 32515 leader_election.cc:302] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148e5a4092f64857a683785cff7be05d, 78cb732272164c69924afb8acad2df2d; no voters:
I20260809 20:28:49.585685 462 raft_consensus.cc:2781] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:28:49.586174 462 raft_consensus.cc:686] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [term 1 LEADER]: Becoming Leader. State: Replica: 148e5a4092f64857a683785cff7be05d, State: Running, Role: LEADER
I20260809 20:28:49.586709 462 consensus_queue.cc:237] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb732272164c69924afb8acad2df2d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37749 } } peers { permanent_uuid: "a337e8fce8f6404a9f0cfef08bf5b03d" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37251 } } peers { permanent_uuid: "148e5a4092f64857a683785cff7be05d" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33417 } }
I20260809 20:28:49.591801 32438 catalog_manager.cc:5495] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d reported cstate change: term changed from 0 to 1, leader changed from <none> to 148e5a4092f64857a683785cff7be05d (127.12.78.1). New cstate: current_term: 1 leader_uuid: "148e5a4092f64857a683785cff7be05d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb732272164c69924afb8acad2df2d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a337e8fce8f6404a9f0cfef08bf5b03d" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148e5a4092f64857a683785cff7be05d" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 33417 } health_report { overall_health: HEALTHY } } }
W20260809 20:28:49.592634 437 tablet.cc:2369] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:28:49.625947 300 tablet.cc:2369] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:28:49.650068 454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.644658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.12.78.1:33417)
0809 20:28:49.644890 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:49.644896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:49.645029 (+ 133us) server_negotiation.cc:407] Connection header received
0809 20:28:49.645215 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:49.645219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:49.645311 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:49.645442 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:28:49.645828 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.646517 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.647901 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.648308 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.648404 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.648933 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:28:49.649420 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:28:49.649555 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:49.649671 (+ 116us) server_negotiation.cc:299] Negotiation successful
0809 20:28:49.649710 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 20:28:49.658443 391 raft_consensus.cc:1261] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d [term 1 FOLLOWER]: Refusing update from remote peer 148e5a4092f64857a683785cff7be05d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:28:49.658445 32721 raft_consensus.cc:1261] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d [term 1 FOLLOWER]: Refusing update from remote peer 148e5a4092f64857a683785cff7be05d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:28:49.659425 467 consensus_queue.cc:1035] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78cb732272164c69924afb8acad2df2d" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 37749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:28:49.659811 462 consensus_queue.cc:1035] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a337e8fce8f6404a9f0cfef08bf5b03d" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 20:28:49.661813 32631 tablet.cc:2369] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:28:49.677148 472 mvcc.cc:205] Tried to move back new op lower bound from 7316714822270582784 to 7316714821991436288. Current Snapshot: MvccSnapshot[applied={T|T < 7316714822270582784 or (T in {7316714822270582784})}]
I20260809 20:28:49.691103 32437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:28:49.691396 32437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:28:49.695114 32437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73dfaea8eb114756b83f262830fc8f23: Sending DeleteTablet for 3 replicas of tablet 5e98acde40db4464872b2b850fa820ff
I20260809 20:28:49.696285 371 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e98acde40db4464872b2b850fa820ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:49 UTC) from {username='slave'} at 127.0.0.1:37408
I20260809 20:28:49.696686 32564 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e98acde40db4464872b2b850fa820ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:49 UTC) from {username='slave'} at 127.0.0.1:56314
I20260809 20:28:49.697358 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 32500
I20260809 20:28:49.697410 32700 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e98acde40db4464872b2b850fa820ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:49 UTC) from {username='slave'} at 127.0.0.1:48540
I20260809 20:28:49.708869 480 tablet_replica.cc:330] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d: stopping tablet replica
I20260809 20:28:49.709014 482 tablet_replica.cc:330] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d: stopping tablet replica
I20260809 20:28:49.709484 482 raft_consensus.cc:2229] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:28:49.709484 480 raft_consensus.cc:2229] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:28:49.710028 482 raft_consensus.cc:2258] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:49.710037 480 raft_consensus.cc:2258] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:49.711635 481 tablet_replica.cc:330] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d: stopping tablet replica
I20260809 20:28:49.712078 481 raft_consensus.cc:2229] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:28:49.712198 482 ts_tablet_manager.cc:1905] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:49.712963 481 raft_consensus.cc:2258] T 5e98acde40db4464872b2b850fa820ff P 148e5a4092f64857a683785cff7be05d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:49.713109 480 ts_tablet_manager.cc:1905] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:49.719152 454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.700025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.12.78.1:33417)
0809 20:28:49.700244 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:49.700253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:49.710141 (+ 9888us) server_negotiation.cc:407] Connection header received
0809 20:28:49.710672 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:49.710681 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:49.710807 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:49.710916 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:49.711945 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.713170 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.715081 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.715728 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.715826 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.717346 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:49.717358 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:49.717374 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:49.718216 (+ 842us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:49.718276 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:49.718303 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:49.718383 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:49.718635 (+ 252us) server_negotiation.cc:299] Negotiation successful
0809 20:28:49.718748 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 20:28:49.719355 482 ts_tablet_manager.cc:1918] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:49.719542 482 log.cc:1196] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/5e98acde40db4464872b2b850fa820ff
I20260809 20:28:49.720386 32625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:49.720428 480 ts_tablet_manager.cc:1918] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:49.721228 482 ts_tablet_manager.cc:1939] T 5e98acde40db4464872b2b850fa820ff P a337e8fce8f6404a9f0cfef08bf5b03d: Deleting consensus metadata
I20260809 20:28:49.721446 480 log.cc:1196] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/5e98acde40db4464872b2b850fa820ff
I20260809 20:28:49.721920 480 ts_tablet_manager.cc:1939] T 5e98acde40db4464872b2b850fa820ff P 78cb732272164c69924afb8acad2df2d: Deleting consensus metadata
I20260809 20:28:49.723498 32421 catalog_manager.cc:4841] TS 78cb732272164c69924afb8acad2df2d (127.12.78.3:37749): tablet 5e98acde40db4464872b2b850fa820ff (table test-table [id=9ead03f10b384d6ca88ba73738f9c09d]) successfully deleted
I20260809 20:28:49.723603 32423 catalog_manager.cc:4841] TS a337e8fce8f6404a9f0cfef08bf5b03d (127.12.78.2:37251): tablet 5e98acde40db4464872b2b850fa820ff (table test-table [id=9ead03f10b384d6ca88ba73738f9c09d]) successfully deleted
I20260809 20:28:49.820115 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 32500
I20260809 20:28:49.841289 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 32635
W20260809 20:28:49.841209 32423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.78.1:33417 (error 108)
I20260809 20:28:49.843976 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.843391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:33417 (local address 127.0.0.1:56340)
0809 20:28:49.843669 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:49.843776 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:33417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260809 20:28:49.844300 32423 catalog_manager.cc:4584] TS 148e5a4092f64857a683785cff7be05d (127.12.78.1:33417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e98acde40db4464872b2b850fa820ff: Network error: Client connection negotiation failed: client connection to 127.12.78.1:33417: connect: Connection refused (error 111)
I20260809 20:28:49.850543 451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.842199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.12.78.2:37251)
0809 20:28:49.842990 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:49.842995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:49.843011 (+ 16us) server_negotiation.cc:407] Connection header received
0809 20:28:49.843064 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:49.843068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:49.843198 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:49.843320 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:49.843910 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.844992 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.847113 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.847533 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.847651 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.849111 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:49.849119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:49.849134 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:49.849712 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:49.849759 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:49.849779 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:49.849873 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:49.850027 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 20:28:49.850115 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642}
I20260809 20:28:49.851246 32762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:49.898121 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.897616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:33417 (local address 127.0.0.1:56342)
0809 20:28:49.897830 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:49.897947 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:33417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 20:28:49.899289 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.898958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:33417 (local address 127.0.0.1:56348)
0809 20:28:49.899096 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:49.899168 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:33417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 20:28:49.943840 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 32635
I20260809 20:28:49.959906 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.959386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:33417 (local address 127.0.0.1:56352)
0809 20:28:49.959635 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:49.959745 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:33417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260809 20:28:49.961134 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.960791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:33417 (local address 127.0.0.1:56362)
0809 20:28:49.960947 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:49.961017 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:33417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 20:28:49.963824 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 302
I20260809 20:28:49.974050 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:49.964593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.12.78.3:37749)
0809 20:28:49.964766 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:49.964770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:49.964789 (+ 19us) server_negotiation.cc:407] Connection header received
0809 20:28:49.964927 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:49.964931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:49.965029 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:49.965129 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:49.965575 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.966328 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.967625 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:49.968020 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:49.968127 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:49.972718 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:49.972728 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:49.972745 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:49.973338 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:49.973396 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:49.973418 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:49.973562 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:49.973690 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 20:28:49.973790 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 20:28:49.974745 431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:50.042107 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:50.041604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:33417 (local address 127.0.0.1:56376)
0809 20:28:50.041831 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:50.041918 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:33417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 20:28:50.043229 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:50.042817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:33417 (local address 127.0.0.1:56390)
0809 20:28:50.043027 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:50.043116 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:33417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260809 20:28:50.065335 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 302
I20260809 20:28:50.084291 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 32404
I20260809 20:28:50.085541 32468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:50.196995 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 32404
2026-08-09T20:28:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2887 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:28:50 dist-test-slave-rnrw krb5kdc[493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:28:50 dist-test-slave-rnrw krb5kdc[493](info): set up 2 sockets
Aug 09 20:28:50 dist-test-slave-rnrw krb5kdc[493](info): commencing operation
krb5kdc: starting...
W20260809 20:28:51.364391 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:28:51 dist-test-slave-rnrw krb5kdc[493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:28:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:28:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:28:51.671586 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40509
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:40509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:52.012526 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:52.012997 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:52.013163 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:52.023639 506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:28:52.023748 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:52.023852 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:52.023941 506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:28:52.024029 506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:28:52.034204 506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:40509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:52.035552 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:52.036972 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:52.049152 512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:52.049188 513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:52.049230 515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:52.050041 506 server_base.cc:1034] running on GCE node
I20260809 20:28:52.050872 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:52.052276 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:52.053507 506 hybrid_clock.cc:648] HybridClock initialized: now 1786307332053431 us; error 67 us; skew 500 ppm
I20260809 20:28:52.053874 506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:52 dist-test-slave-rnrw krb5kdc[493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307332, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:52.056973 506 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:28:52.058638 506 webserver.cc:460] Webserver started at http://127.12.78.62:33035/ using document root <none> and password file <none>
I20260809 20:28:52.059304 506 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:52.059449 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:52.059762 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:52.062774 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "2e52227b34bf4b71baee4e06f7a3a14a"
format_stamp: "Formatted at 2026-08-09 20:28:52 on dist-test-slave-rnrw"
server_key: "87530c60088ad634ad77ce054729a53b"
server_key_iv: "8876705f434f91e659b3ad50c016176a"
server_key_version: "encryptionkey@0"
I20260809 20:28:52.063524 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "2e52227b34bf4b71baee4e06f7a3a14a"
format_stamp: "Formatted at 2026-08-09 20:28:52 on dist-test-slave-rnrw"
server_key: "87530c60088ad634ad77ce054729a53b"
server_key_iv: "8876705f434f91e659b3ad50c016176a"
server_key_version: "encryptionkey@0"
I20260809 20:28:52.068531 506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 20:28:52.071879 523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:52.073220 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:28:52.073410 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "2e52227b34bf4b71baee4e06f7a3a14a"
format_stamp: "Formatted at 2026-08-09 20:28:52 on dist-test-slave-rnrw"
server_key: "87530c60088ad634ad77ce054729a53b"
server_key_iv: "8876705f434f91e659b3ad50c016176a"
server_key_version: "encryptionkey@0"
I20260809 20:28:52.073621 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:52.096323 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:52.100473 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:52.100793 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:52.119573 506 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:40509
I20260809 20:28:52.119617 574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:40509 every 8 connection(s)
I20260809 20:28:52.120970 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:28:52.124647 575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:52.130293 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 506
I20260809 20:28:52.130479 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:28:52.130919 12600 external_mini_cluster.cc:1428] Setting key ad79264a22a0fc1e875de42f6d038f11
I20260809 20:28:52.135658 575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a: Bootstrap starting.
I20260809 20:28:52.139416 575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:52.140475 575 log.cc:824] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:52.143502 575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a: No bootstrap required, opened a new log
Aug 09 20:28:52 dist-test-slave-rnrw krb5kdc[493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:28:52.153066 575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e52227b34bf4b71baee4e06f7a3a14a" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40509 } }
I20260809 20:28:52.153467 575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:52.153597 575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e52227b34bf4b71baee4e06f7a3a14a, State: Initialized, Role: FOLLOWER
I20260809 20:28:52.154207 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:52.132863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59098 (local address 127.12.78.62:40509)
0809 20:28:52.133703 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:52.133717 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:52.133760 (+ 43us) server_negotiation.cc:407] Connection header received
0809 20:28:52.135405 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:52.135426 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:52.135836 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:52.136167 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:52.138124 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:52.139391 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:52.140941 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:52.141723 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:52.141912 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:52.146505 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:52.146515 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:52.146532 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:52.146569 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:52.149842 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:52.150685 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:52.150691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:52.150696 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:52.150890 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:52.151352 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:52.151356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:52.151361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:52.151854 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:52.152100 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:52.152608 (+ 508us) server_negotiation.cc:299] Negotiation successful
0809 20:28:52.153001 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":622,"thread_start_us":394,"threads_started":1}
I20260809 20:28:52.154174 575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e52227b34bf4b71baee4e06f7a3a14a" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40509 } }
I20260809 20:28:52.154500 575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:28:52.154664 575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:28:52.154853 575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:52.157352 575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e52227b34bf4b71baee4e06f7a3a14a" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40509 } }
I20260809 20:28:52.157879 575 leader_election.cc:302] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e52227b34bf4b71baee4e06f7a3a14a; no voters:
I20260809 20:28:52.158476 575 leader_election.cc:288] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:28:52.158608 580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:28:52.159325 580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [term 1 LEADER]: Becoming Leader. State: Replica: 2e52227b34bf4b71baee4e06f7a3a14a, State: Running, Role: LEADER
I20260809 20:28:52.159845 580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e52227b34bf4b71baee4e06f7a3a14a" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40509 } }
I20260809 20:28:52.160688 575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:28:52.162671 581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e52227b34bf4b71baee4e06f7a3a14a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e52227b34bf4b71baee4e06f7a3a14a" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40509 } } }
I20260809 20:28:52.163188 581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:52.163012 582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e52227b34bf4b71baee4e06f7a3a14a. Latest consensus state: current_term: 1 leader_uuid: "2e52227b34bf4b71baee4e06f7a3a14a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e52227b34bf4b71baee4e06f7a3a14a" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40509 } } }
I20260809 20:28:52.163388 582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:52.164920 586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:28:52.170821 586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:28:52.180509 586 catalog_manager.cc:1347] Generated new cluster ID: 5396e9a70d314e99a06c62e2413d5ca1
I20260809 20:28:52.180645 586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:28:52.207240 586 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:28:52.209228 586 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:28:52.238174 586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a: Generated new TSK 0
I20260809 20:28:52.239274 586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.78.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
Entry for principal kudu/127.12.78.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab.
I20260809 20:28:52.321535 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40509
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:52.680001 603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:52.680302 603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:52.680464 603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:52.687944 603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:52.688071 603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:52.688277 603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:28:52.698454 603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.1.keytab
--principal=kudu/127.12.78.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:52.699847 603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:52.701288 603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:52.715384 612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:52.715356 609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:52.715448 608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:52.716141 603 server_base.cc:1034] running on GCE node
I20260809 20:28:52.716908 603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:52.717905 603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:52.719143 603 hybrid_clock.cc:648] HybridClock initialized: now 1786307332719095 us; error 42 us; skew 500 ppm
I20260809 20:28:52.719494 603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:52 dist-test-slave-rnrw krb5kdc[493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307332, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:52.722957 603 init.cc:377] Logged in from keytab as kudu/127.12.78.1@KRBTEST.COM (short username kudu)
I20260809 20:28:52.724686 603 webserver.cc:460] Webserver started at http://127.12.78.1:40493/ using document root <none> and password file <none>
I20260809 20:28:52.725418 603 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:52.725530 603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:52.725832 603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:52.729090 603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd"
format_stamp: "Formatted at 2026-08-09 20:28:52 on dist-test-slave-rnrw"
server_key: "91a1cae14cb1fadbf0d7d0c19c0e6344"
server_key_iv: "f0178a7825f758dea4659d7fbf3e502e"
server_key_version: "encryptionkey@0"
I20260809 20:28:52.729903 603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd"
format_stamp: "Formatted at 2026-08-09 20:28:52 on dist-test-slave-rnrw"
server_key: "91a1cae14cb1fadbf0d7d0c19c0e6344"
server_key_iv: "f0178a7825f758dea4659d7fbf3e502e"
server_key_version: "encryptionkey@0"
I20260809 20:28:52.736152 603 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260809 20:28:52.740291 619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:52.741950 603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:28:52.742165 603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd"
format_stamp: "Formatted at 2026-08-09 20:28:52 on dist-test-slave-rnrw"
server_key: "91a1cae14cb1fadbf0d7d0c19c0e6344"
server_key_iv: "f0178a7825f758dea4659d7fbf3e502e"
server_key_version: "encryptionkey@0"
I20260809 20:28:52.742419 603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:52.788077 603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:52.794322 603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:52.794624 603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:52.795606 603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:52.797115 603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:52.797201 603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:52.797293 603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:52.797338 603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:52.831169 603 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:37133
I20260809 20:28:52.831212 739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:37133 every 8 connection(s)
I20260809 20:28:52.832672 603 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:28:52.834987 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 603
I20260809 20:28:52.835208 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:28:52.835788 12600 external_mini_cluster.cc:1428] Setting key bb8be0cb669bd0f1dafdfaebb624496e
Aug 09 20:28:52 dist-test-slave-rnrw krb5kdc[493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307332, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.1@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:28:52.856583 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:52.835836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:53519 (local address 127.12.78.62:40509)
0809 20:28:52.836362 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:52.836369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:52.837567 (+ 1198us) server_negotiation.cc:407] Connection header received
0809 20:28:52.839577 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:52.839584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:52.839676 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:52.839812 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:52.841558 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:52.842374 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:52.844043 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:52.844522 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:52.844732 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:52.849129 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:52.849135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:52.849139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:52.849177 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:52.852634 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:52.853569 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:52.853579 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:52.853585 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:52.853761 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:52.854298 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:52.854306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:52.854312 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:52.854740 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:52.854905 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:52.855749 (+ 844us) server_negotiation.cc:299] Negotiation successful
0809 20:28:52.856035 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":253,"threads_started":1}
I20260809 20:28:52.857331 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:52.836511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40509 (local address 127.12.78.1:53519)
0809 20:28:52.837412 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:52.837452 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:52.839279 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:52.840078 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:52.840087 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:52.840743 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:52.841359 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:52.841378 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:52.842552 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:52.842562 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:52.843872 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:52.843886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:52.844857 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:52.844869 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:52.845165 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:52.845987 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:28:52.846045 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:52.848839 (+ 2794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:52.852828 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:52.852836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:52.852853 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:52.853397 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:52.853895 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:52.853899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:52.853901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:52.854163 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:52.854897 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:52.854904 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:52.855298 (+ 394us) client_negotiation.cc:769] Sending connection context
0809 20:28:52.855546 (+ 248us) client_negotiation.cc:240] Negotiation successful
0809 20:28:52.856004 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":350,"threads_started":1}
I20260809 20:28:52.859184 740 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40509
I20260809 20:28:52.859565 740 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:52.860687 740 heartbeater.cc:511] Master 127.12.78.62:40509 requested a full tablet report, sending...
I20260809 20:28:52.863277 540 ts_manager.cc:194] Registered new tserver with Master: f0fa29b05f2d47f89ff2b5c6e27348bd (127.12.78.1:37133)
I20260809 20:28:52.864892 540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.1@KRBTEST.COM'} at 127.12.78.1:53519
WARNING: no policy specified for kudu/127.12.78.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
Entry for principal kudu/127.12.78.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab.
I20260809 20:28:52.909004 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40509
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:53.266327 746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:53.266657 746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:53.266844 746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:53.274681 746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:53.274818 746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:53.275012 746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:28:53.285420 746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.2.keytab
--principal=kudu/127.12.78.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:53.286816 746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:53.288283 746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:53.302435 751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:53.302397 752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:53.302522 754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:53.303393 746 server_base.cc:1034] running on GCE node
I20260809 20:28:53.304124 746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:53.305212 746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:53.306438 746 hybrid_clock.cc:648] HybridClock initialized: now 1786307333306362 us; error 65 us; skew 500 ppm
I20260809 20:28:53.306808 746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:53 dist-test-slave-rnrw krb5kdc[493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307333, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:53.310716 746 init.cc:377] Logged in from keytab as kudu/127.12.78.2@KRBTEST.COM (short username kudu)
I20260809 20:28:53.312288 746 webserver.cc:460] Webserver started at http://127.12.78.2:33375/ using document root <none> and password file <none>
I20260809 20:28:53.313135 746 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:53.313253 746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:53.313558 746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:53.316681 746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "f8cd065eb45f4275846881b6f937973c"
format_stamp: "Formatted at 2026-08-09 20:28:53 on dist-test-slave-rnrw"
server_key: "1bf3ff2a1d6053536a82f3693eef6c4c"
server_key_iv: "3417fb76e4988990525735796a6fae53"
server_key_version: "encryptionkey@0"
I20260809 20:28:53.317472 746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "f8cd065eb45f4275846881b6f937973c"
format_stamp: "Formatted at 2026-08-09 20:28:53 on dist-test-slave-rnrw"
server_key: "1bf3ff2a1d6053536a82f3693eef6c4c"
server_key_iv: "3417fb76e4988990525735796a6fae53"
server_key_version: "encryptionkey@0"
I20260809 20:28:53.323040 746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 20:28:53.327881 762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:53.329501 746 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 20:28:53.329663 746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "f8cd065eb45f4275846881b6f937973c"
format_stamp: "Formatted at 2026-08-09 20:28:53 on dist-test-slave-rnrw"
server_key: "1bf3ff2a1d6053536a82f3693eef6c4c"
server_key_iv: "3417fb76e4988990525735796a6fae53"
server_key_version: "encryptionkey@0"
I20260809 20:28:53.329851 746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:53.350262 746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:53.354627 746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:53.354888 746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:53.356050 746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:53.357620 746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:53.357702 746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:53.357820 746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:53.357887 746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:53.392814 746 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:34537
I20260809 20:28:53.392858 881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:34537 every 8 connection(s)
I20260809 20:28:53.394186 746 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:28:53.400843 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 746
I20260809 20:28:53.401088 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:28:53.401674 12600 external_mini_cluster.cc:1428] Setting key 31d9d500374a797940a8d94314c54666
Aug 09 20:28:53 dist-test-slave-rnrw krb5kdc[493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307333, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.2@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:28:53.418036 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:53.397122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:60823 (local address 127.12.78.62:40509)
0809 20:28:53.397557 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:53.397563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:53.398647 (+ 1084us) server_negotiation.cc:407] Connection header received
0809 20:28:53.400531 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:53.400537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:53.400689 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:53.400822 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:53.402596 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:53.403548 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:53.405270 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:53.406164 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:53.406348 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:53.410343 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:53.410353 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:53.410361 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:53.410412 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:53.414618 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:53.415427 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:53.415437 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:53.415443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:53.415631 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:53.416068 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:53.416076 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:53.416082 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:53.416523 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:53.416743 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:53.417506 (+ 763us) server_negotiation.cc:299] Negotiation successful
0809 20:28:53.417708 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":210,"threads_started":1}
I20260809 20:28:53.419557 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:53.397746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40509 (local address 127.12.78.2:60823)
0809 20:28:53.398470 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:53.398510 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:53.400240 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:53.401116 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:53.401124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:53.401695 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:53.402392 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:53.402419 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:53.403717 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:53.403722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:53.405111 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:53.405124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:53.406428 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:53.406433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:53.406687 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:53.407423 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:28:53.407463 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:53.410053 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:53.414808 (+ 4755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:53.414816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:53.414831 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:53.415291 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:53.415757 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:53.415760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:53.415763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:53.415951 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:53.416712 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:53.416719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:53.417073 (+ 354us) client_negotiation.cc:769] Sending connection context
0809 20:28:53.417398 (+ 325us) client_negotiation.cc:240] Negotiation successful
0809 20:28:53.418041 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":288,"threads_started":1}
I20260809 20:28:53.421146 882 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40509
I20260809 20:28:53.421545 882 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:53.422369 882 heartbeater.cc:511] Master 127.12.78.62:40509 requested a full tablet report, sending...
I20260809 20:28:53.424036 540 ts_manager.cc:194] Registered new tserver with Master: f8cd065eb45f4275846881b6f937973c (127.12.78.2:34537)
I20260809 20:28:53.425064 540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.2@KRBTEST.COM'} at 127.12.78.2:60823
WARNING: no policy specified for kudu/127.12.78.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
Entry for principal kudu/127.12.78.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab.
I20260809 20:28:53.472169 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40509
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:53.822471 888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:53.822778 888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:53.822955 888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:53.830307 888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:53.830495 888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:53.830760 888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:28:53.841249 888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.3.keytab
--principal=kudu/127.12.78.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:53.842648 888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:53.844043 888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:53.858224 893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:53.858295 894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:53.858268 896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:53.860419 888 server_base.cc:1034] running on GCE node
I20260809 20:28:53.861263 888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:53.862361 888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:53.863619 888 hybrid_clock.cc:648] HybridClock initialized: now 1786307333863519 us; error 75 us; skew 500 ppm
I20260809 20:28:53.863966 888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:53 dist-test-slave-rnrw krb5kdc[493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307333, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:53.867460 888 init.cc:377] Logged in from keytab as kudu/127.12.78.3@KRBTEST.COM (short username kudu)
I20260809 20:28:53.869251 740 heartbeater.cc:503] Master 127.12.78.62:40509 was elected leader, sending a full tablet report...
I20260809 20:28:53.869330 888 webserver.cc:460] Webserver started at http://127.12.78.3:43901/ using document root <none> and password file <none>
I20260809 20:28:53.870224 888 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:53.870362 888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:53.870707 888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:53.873867 888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "5fc41541bd6645bba1e2a6b922d443e0"
format_stamp: "Formatted at 2026-08-09 20:28:53 on dist-test-slave-rnrw"
server_key: "3da1b76154eea1dcf2cc7018420dba42"
server_key_iv: "fd1c5a3a96705318964c66e1975550da"
server_key_version: "encryptionkey@0"
I20260809 20:28:53.874668 888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "5fc41541bd6645bba1e2a6b922d443e0"
format_stamp: "Formatted at 2026-08-09 20:28:53 on dist-test-slave-rnrw"
server_key: "3da1b76154eea1dcf2cc7018420dba42"
server_key_iv: "fd1c5a3a96705318964c66e1975550da"
server_key_version: "encryptionkey@0"
I20260809 20:28:53.880301 888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 20:28:53.883965 903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:53.885360 888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:28:53.885537 888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "5fc41541bd6645bba1e2a6b922d443e0"
format_stamp: "Formatted at 2026-08-09 20:28:53 on dist-test-slave-rnrw"
server_key: "3da1b76154eea1dcf2cc7018420dba42"
server_key_iv: "fd1c5a3a96705318964c66e1975550da"
server_key_version: "encryptionkey@0"
I20260809 20:28:53.885749 888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:53.923980 888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:53.928534 888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:53.928897 888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:53.929919 888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:53.931291 888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:53.931404 888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:53.931545 888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:53.931620 888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:53.965999 888 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:34841
I20260809 20:28:53.966046 1023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:34841 every 8 connection(s)
I20260809 20:28:53.967757 888 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:28:53.974783 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 888
I20260809 20:28:53.975003 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:28:53.975498 12600 external_mini_cluster.cc:1428] Setting key 178b9d4b7ec48bf6d8e65a3268279068
Aug 09 20:28:53 dist-test-slave-rnrw krb5kdc[493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307333, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.3@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:28:53.988417 1026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:53.970284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:48497 (local address 127.12.78.62:40509)
0809 20:28:53.970769 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:53.970775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:53.971728 (+ 953us) server_negotiation.cc:407] Connection header received
0809 20:28:53.973656 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:53.973661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:53.973746 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:53.973851 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:53.975503 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:53.976501 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:53.978516 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:53.979011 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:53.979125 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:53.983117 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:53.983122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:53.983125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:53.983160 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:53.985683 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:53.986408 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:53.986412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:53.986415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:53.986533 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:53.986901 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:53.986904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:53.986906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:53.987186 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:53.987297 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:53.987952 (+ 655us) server_negotiation.cc:299] Negotiation successful
0809 20:28:53.988138 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":219,"threads_started":1}
I20260809 20:28:53.989852 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:53.970888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40509 (local address 127.12.78.3:48497)
0809 20:28:53.971557 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:53.971594 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:53.973391 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:53.974060 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:53.974069 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:53.974601 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:53.975304 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:53.975330 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:53.976724 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:53.976728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:53.978329 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:53.978342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:53.979233 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:53.979238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:53.979443 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:53.980110 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:28:53.980134 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:53.982885 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:53.985821 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:53.985828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:53.985842 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:53.986297 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:53.986624 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:28:53.986627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:28:53.986629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:28:53.986810 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:28:53.987290 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:53.987296 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:53.987630 (+ 334us) client_negotiation.cc:769] Sending connection context
0809 20:28:53.987865 (+ 235us) client_negotiation.cc:240] Negotiation successful
0809 20:28:53.988297 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":282,"threads_started":1}
I20260809 20:28:53.991402 1024 heartbeater.cc:347] Connected to a master server at 127.12.78.62:40509
I20260809 20:28:53.991724 1024 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:53.992446 1024 heartbeater.cc:511] Master 127.12.78.62:40509 requested a full tablet report, sending...
I20260809 20:28:53.993880 540 ts_manager.cc:194] Registered new tserver with Master: 5fc41541bd6645bba1e2a6b922d443e0 (127.12.78.3:34841)
I20260809 20:28:53.994706 540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.78.3@KRBTEST.COM'} at 127.12.78.3:48497
I20260809 20:28:54.004142 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:28:54.021533 1026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.008475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.12.78.62:40509)
0809 20:28:54.008759 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:54.008766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:54.009112 (+ 346us) server_negotiation.cc:407] Connection header received
0809 20:28:54.009342 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:54.009346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:54.009413 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:54.009523 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:54.011004 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.011739 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.013161 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.013574 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.013715 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.016126 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:54.016131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:54.016134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:54.016169 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:54.018747 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:54.019421 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:54.019427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:54.019431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:54.019562 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:54.019951 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:54.019956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:54.019960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:54.020325 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:54.020626 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:54.020954 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 20:28:54.021211 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260809 20:28:54.025722 540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:28:54.032806 540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:28:54.061291 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.047698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.12.78.1:37133)
0809 20:28:54.049809 (+ 2111us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:54.049820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:54.050489 (+ 669us) server_negotiation.cc:407] Connection header received
0809 20:28:54.050684 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:54.050695 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:54.050944 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:54.051152 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:54.052213 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.054025 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.057544 (+ 3519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.059371 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.059538 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.059808 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:54.060673 (+ 865us) server_negotiation.cc:299] Negotiation successful
0809 20:28:54.060883 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1854,"thread_start_us":1686,"threads_started":1}
I20260809 20:28:54.061704 1037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.046704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:34537 (local address 127.0.0.1:54964)
0809 20:28:54.048498 (+ 1794us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:54.048573 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:54.048750 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:54.049901 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:54.049910 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:54.049944 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:54.050507 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:54.050526 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.054083 (+ 3557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.054092 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:54.057324 (+ 3232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:54.057342 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.060678 (+ 3336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.060686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:54.060870 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.060903 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 20:28:54.060989 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 20:28:54.061221 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1536,"mutex_wait_us":99,"thread_start_us":1388,"threads_started":1}
I20260809 20:28:54.062934 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.047046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:37133 (local address 127.0.0.1:54150)
0809 20:28:54.050345 (+ 3299us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:54.050363 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:54.050496 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:54.051426 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:54.051432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:54.051458 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:54.051989 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:54.052006 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.054190 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.054197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:54.057331 (+ 3134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:54.057342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.059998 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.060005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:54.060278 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.060404 (+ 126us) client_negotiation.cc:769] Sending connection context
0809 20:28:54.062297 (+ 1893us) client_negotiation.cc:240] Negotiation successful
0809 20:28:54.062457 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3152,"mutex_wait_us":204,"thread_start_us":2988,"threads_started":1}
I20260809 20:28:54.063191 1038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.046991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.12.78.2:34537)
0809 20:28:54.049285 (+ 2294us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:54.049292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:54.049316 (+ 24us) server_negotiation.cc:407] Connection header received
0809 20:28:54.049395 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:54.049402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:54.049603 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:54.050779 (+ 1176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:54.051036 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.053881 (+ 2845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.057544 (+ 3663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.060394 (+ 2850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.062295 (+ 1901us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.062521 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:54.062603 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 20:28:54.062795 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2159,"thread_start_us":2039,"threads_started":1}
I20260809 20:28:54.067726 813 tablet_service.cc:1467] Processing CreateTablet for tablet bdd5516391f64317b65a8197c179b31e (DEFAULT_TABLE table=test-table [id=3680ca5ee786437b87484f3c422a07b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:54.068516 672 tablet_service.cc:1467] Processing CreateTablet for tablet bdd5516391f64317b65a8197c179b31e (DEFAULT_TABLE table=test-table [id=3680ca5ee786437b87484f3c422a07b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:54.069289 813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd5516391f64317b65a8197c179b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:54.070003 672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd5516391f64317b65a8197c179b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:54.070715 1037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.050422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:34841 (local address 127.0.0.1:56336)
0809 20:28:54.061955 (+ 11533us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:54.061990 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:54.062102 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:54.064436 (+ 2334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:54.064442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:54.064482 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:54.064731 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:54.064746 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.066867 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.066873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:54.068240 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:54.068253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.070112 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.070119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:54.070290 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.070317 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 20:28:54.070376 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 20:28:54.070495 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11422,"thread_start_us":14464,"threads_started":1}
I20260809 20:28:54.072044 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.051351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.12.78.3:34841)
0809 20:28:54.063878 (+ 12527us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:54.063885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:54.063917 (+ 32us) server_negotiation.cc:407] Connection header received
0809 20:28:54.064012 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:54.064018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:54.064227 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:54.064457 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:54.064970 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.066710 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.068503 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.069878 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.071127 (+ 1249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.071381 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:54.071460 (+ 79us) server_negotiation.cc:299] Negotiation successful
0809 20:28:54.071643 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12366,"thread_start_us":14883,"threads_started":1}
I20260809 20:28:54.075408 951 tablet_service.cc:1467] Processing CreateTablet for tablet bdd5516391f64317b65a8197c179b31e (DEFAULT_TABLE table=test-table [id=3680ca5ee786437b87484f3c422a07b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:54.076860 951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd5516391f64317b65a8197c179b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:54.084764 1044 tablet_bootstrap.cc:492] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c: Bootstrap starting.
I20260809 20:28:54.085661 1043 tablet_bootstrap.cc:492] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd: Bootstrap starting.
I20260809 20:28:54.089463 1044 tablet_bootstrap.cc:654] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:54.089704 1043 tablet_bootstrap.cc:654] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:54.090844 1044 log.cc:824] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:54.091248 1045 tablet_bootstrap.cc:492] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0: Bootstrap starting.
I20260809 20:28:54.091181 1043 log.cc:824] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:54.094646 1044 tablet_bootstrap.cc:492] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c: No bootstrap required, opened a new log
I20260809 20:28:54.094825 1043 tablet_bootstrap.cc:492] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd: No bootstrap required, opened a new log
I20260809 20:28:54.094998 1044 ts_tablet_manager.cc:1397] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260809 20:28:54.095109 1043 ts_tablet_manager.cc:1397] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 20:28:54.095477 1045 tablet_bootstrap.cc:654] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:54.096760 1045 log.cc:824] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:54.100178 1045 tablet_bootstrap.cc:492] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0: No bootstrap required, opened a new log
I20260809 20:28:54.100478 1045 ts_tablet_manager.cc:1397] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260809 20:28:54.103657 1043 raft_consensus.cc:348] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cd065eb45f4275846881b6f937973c" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 34537 } } peers { permanent_uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 37133 } } peers { permanent_uuid: "5fc41541bd6645bba1e2a6b922d443e0" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34841 } }
I20260809 20:28:54.104043 1043 raft_consensus.cc:374] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:54.104149 1043 raft_consensus.cc:729] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0fa29b05f2d47f89ff2b5c6e27348bd, State: Initialized, Role: FOLLOWER
I20260809 20:28:54.104795 1043 consensus_queue.cc:260] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cd065eb45f4275846881b6f937973c" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 34537 } } peers { permanent_uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 37133 } } peers { permanent_uuid: "5fc41541bd6645bba1e2a6b922d443e0" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34841 } }
I20260809 20:28:54.106755 1043 ts_tablet_manager.cc:1428] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260809 20:28:54.106621 1044 raft_consensus.cc:348] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cd065eb45f4275846881b6f937973c" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 34537 } } peers { permanent_uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 37133 } } peers { permanent_uuid: "5fc41541bd6645bba1e2a6b922d443e0" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34841 } }
I20260809 20:28:54.107077 1044 raft_consensus.cc:374] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:54.107199 1044 raft_consensus.cc:729] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cd065eb45f4275846881b6f937973c, State: Initialized, Role: FOLLOWER
I20260809 20:28:54.107811 1044 consensus_queue.cc:260] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cd065eb45f4275846881b6f937973c" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 34537 } } peers { permanent_uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 37133 } } peers { permanent_uuid: "5fc41541bd6645bba1e2a6b922d443e0" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34841 } }
I20260809 20:28:54.109015 882 heartbeater.cc:503] Master 127.12.78.62:40509 was elected leader, sending a full tablet report...
I20260809 20:28:54.109654 1044 ts_tablet_manager.cc:1428] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260809 20:28:54.112538 1045 raft_consensus.cc:348] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cd065eb45f4275846881b6f937973c" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 34537 } } peers { permanent_uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 37133 } } peers { permanent_uuid: "5fc41541bd6645bba1e2a6b922d443e0" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34841 } }
I20260809 20:28:54.113076 1045 raft_consensus.cc:374] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:54.113196 1045 raft_consensus.cc:729] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc41541bd6645bba1e2a6b922d443e0, State: Initialized, Role: FOLLOWER
I20260809 20:28:54.113850 1045 consensus_queue.cc:260] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cd065eb45f4275846881b6f937973c" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 34537 } } peers { permanent_uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 37133 } } peers { permanent_uuid: "5fc41541bd6645bba1e2a6b922d443e0" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34841 } }
I20260809 20:28:54.114930 1024 heartbeater.cc:503] Master 127.12.78.62:40509 was elected leader, sending a full tablet report...
I20260809 20:28:54.115427 1045 ts_tablet_manager.cc:1428] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
W20260809 20:28:54.146304 883 tablet.cc:2369] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 20:28:54.219520 1025 tablet.cc:2369] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:28:54.253414 1051 raft_consensus.cc:482] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:28:54.253698 1051 raft_consensus.cc:504] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cd065eb45f4275846881b6f937973c" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 34537 } } peers { permanent_uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 37133 } } peers { permanent_uuid: "5fc41541bd6645bba1e2a6b922d443e0" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34841 } }
I20260809 20:28:54.255363 1051 leader_election.cc:288] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8cd065eb45f4275846881b6f937973c (127.12.78.2:34537), f0fa29b05f2d47f89ff2b5c6e27348bd (127.12.78.1:37133)
I20260809 20:28:54.262566 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.255854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:45241 (local address 127.12.78.1:37133)
0809 20:28:54.256080 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:54.256086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:54.256114 (+ 28us) server_negotiation.cc:407] Connection header received
0809 20:28:54.256208 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:54.256217 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:54.256322 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:54.256434 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:54.256945 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.257982 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.260312 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.261516 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.261676 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.261790 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:54.262181 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 20:28:54.262312 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 20:28:54.262598 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.255650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:37133 (local address 127.12.78.3:45241)
0809 20:28:54.255911 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:54.255933 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:54.256070 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:54.256488 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:54.256500 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:54.256558 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:54.256807 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:54.256819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.258122 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.258126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:54.260141 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:54.260156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.261770 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.261774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:54.261941 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.261967 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 20:28:54.262112 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 20:28:54.262259 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":31}
I20260809 20:28:54.263144 1038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.255974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:56393 (local address 127.12.78.2:34537)
0809 20:28:54.256181 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:54.256189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:54.256795 (+ 606us) server_negotiation.cc:407] Connection header received
0809 20:28:54.256941 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:54.256946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:54.257031 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:54.257139 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:54.257595 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.258460 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.260793 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.262094 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.262237 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.262344 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:54.262654 (+ 310us) server_negotiation.cc:299] Negotiation successful
0809 20:28:54.262890 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 20:28:54.263187 1052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.255648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:34537 (local address 127.12.78.3:56393)
0809 20:28:54.256686 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:54.256702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:54.256819 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:54.257155 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:54.257158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:54.257177 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:54.257452 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:54.257459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.258610 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.258615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:54.260651 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:54.260663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.262299 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.262305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:54.262481 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.262505 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 20:28:54.262627 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 20:28:54.262754 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":366,"thread_start_us":343,"threads_started":1}
I20260809 20:28:54.263999 692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd5516391f64317b65a8197c179b31e" candidate_uuid: "5fc41541bd6645bba1e2a6b922d443e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd" is_pre_election: true
I20260809 20:28:54.264220 834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd5516391f64317b65a8197c179b31e" candidate_uuid: "5fc41541bd6645bba1e2a6b922d443e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cd065eb45f4275846881b6f937973c" is_pre_election: true
I20260809 20:28:54.264606 692 raft_consensus.cc:2454] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fc41541bd6645bba1e2a6b922d443e0 in term 0.
I20260809 20:28:54.264822 834 raft_consensus.cc:2454] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fc41541bd6645bba1e2a6b922d443e0 in term 0.
I20260809 20:28:54.265300 907 leader_election.cc:302] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fc41541bd6645bba1e2a6b922d443e0, f0fa29b05f2d47f89ff2b5c6e27348bd; no voters:
I20260809 20:28:54.265734 1051 raft_consensus.cc:2781] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:28:54.265894 1051 raft_consensus.cc:482] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:28:54.266013 1051 raft_consensus.cc:3037] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:54.268364 1051 raft_consensus.cc:504] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cd065eb45f4275846881b6f937973c" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 34537 } } peers { permanent_uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 37133 } } peers { permanent_uuid: "5fc41541bd6645bba1e2a6b922d443e0" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34841 } }
I20260809 20:28:54.269048 1051 leader_election.cc:288] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [CANDIDATE]: Term 1 election: Requested vote from peers f8cd065eb45f4275846881b6f937973c (127.12.78.2:34537), f0fa29b05f2d47f89ff2b5c6e27348bd (127.12.78.1:37133)
I20260809 20:28:54.269510 834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd5516391f64317b65a8197c179b31e" candidate_uuid: "5fc41541bd6645bba1e2a6b922d443e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cd065eb45f4275846881b6f937973c"
I20260809 20:28:54.269624 692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd5516391f64317b65a8197c179b31e" candidate_uuid: "5fc41541bd6645bba1e2a6b922d443e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd"
I20260809 20:28:54.269770 834 raft_consensus.cc:3037] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:54.269850 692 raft_consensus.cc:3037] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:54.273643 692 raft_consensus.cc:2454] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fc41541bd6645bba1e2a6b922d443e0 in term 1.
I20260809 20:28:54.273643 834 raft_consensus.cc:2454] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fc41541bd6645bba1e2a6b922d443e0 in term 1.
I20260809 20:28:54.274181 907 leader_election.cc:302] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fc41541bd6645bba1e2a6b922d443e0, f0fa29b05f2d47f89ff2b5c6e27348bd; no voters:
I20260809 20:28:54.274524 1051 raft_consensus.cc:2781] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:28:54.275009 1051 raft_consensus.cc:686] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc41541bd6645bba1e2a6b922d443e0, State: Running, Role: LEADER
I20260809 20:28:54.275601 1051 consensus_queue.cc:237] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cd065eb45f4275846881b6f937973c" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 34537 } } peers { permanent_uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 37133 } } peers { permanent_uuid: "5fc41541bd6645bba1e2a6b922d443e0" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34841 } }
I20260809 20:28:54.280861 539 catalog_manager.cc:5495] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fc41541bd6645bba1e2a6b922d443e0 (127.12.78.3). New cstate: current_term: 1 leader_uuid: "5fc41541bd6645bba1e2a6b922d443e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cd065eb45f4275846881b6f937973c" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 34537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 37133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fc41541bd6645bba1e2a6b922d443e0" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 34841 } health_report { overall_health: HEALTHY } } }
I20260809 20:28:54.299649 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.293905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.12.78.3:34841)
0809 20:28:54.294107 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:54.294116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:54.294174 (+ 58us) server_negotiation.cc:407] Connection header received
0809 20:28:54.294420 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:54.294424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:54.294514 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:54.294675 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:28:54.295056 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.295776 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.297263 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.297880 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.298024 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.298541 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:28:54.299058 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:28:54.299224 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:54.299283 (+ 59us) server_negotiation.cc:299] Negotiation successful
0809 20:28:54.299368 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 20:28:54.308007 834 raft_consensus.cc:1261] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c [term 1 FOLLOWER]: Refusing update from remote peer 5fc41541bd6645bba1e2a6b922d443e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:28:54.308066 692 raft_consensus.cc:1261] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd [term 1 FOLLOWER]: Refusing update from remote peer 5fc41541bd6645bba1e2a6b922d443e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:28:54.309069 1051 consensus_queue.cc:1035] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0fa29b05f2d47f89ff2b5c6e27348bd" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 37133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:28:54.309407 1053 consensus_queue.cc:1035] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8cd065eb45f4275846881b6f937973c" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:28:54.322494 1058 mvcc.cc:205] Tried to move back new op lower bound from 7316714841314902016 to 7316714841197244416. Current Snapshot: MvccSnapshot[applied={T|T < 7316714841314902016}]
I20260809 20:28:54.335696 540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:28:54.335991 540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:28:54.339689 540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e52227b34bf4b71baee4e06f7a3a14a: Sending DeleteTablet for 3 replicas of tablet bdd5516391f64317b65a8197c179b31e
I20260809 20:28:54.341023 813 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd5516391f64317b65a8197c179b31e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:54 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:54964
I20260809 20:28:54.341171 672 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd5516391f64317b65a8197c179b31e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:54 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:54150
I20260809 20:28:54.341898 951 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd5516391f64317b65a8197c179b31e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:54 UTC) from {username='kudu', principal='kudu/127.12.78.62@KRBTEST.COM'} at 127.0.0.1:56336
I20260809 20:28:54.342185 1065 tablet_replica.cc:330] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c: stopping tablet replica
I20260809 20:28:54.342360 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 603
I20260809 20:28:54.342686 1065 raft_consensus.cc:2229] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:28:54.343307 1065 raft_consensus.cc:2258] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:54.347746 1065 ts_tablet_manager.cc:1905] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:54.348214 1067 tablet_replica.cc:330] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0: stopping tablet replica
I20260809 20:28:54.348723 1067 raft_consensus.cc:2229] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [term 1 LEADER]: Raft consensus shutting down.
I20260809 20:28:54.349327 1066 tablet_replica.cc:330] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd: stopping tablet replica
I20260809 20:28:54.349756 1066 raft_consensus.cc:2229] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:28:54.350301 1066 raft_consensus.cc:2258] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:54.351207 1067 raft_consensus.cc:2258] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:54.352839 1066 ts_tablet_manager.cc:1905] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 20:28:54 dist-test-slave-rnrw krb5kdc[493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.1@KRBTEST.COM
I20260809 20:28:54.357578 1067 ts_tablet_manager.cc:1905] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:54.358487 1065 ts_tablet_manager.cc:1918] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:54.358673 1065 log.cc:1196] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/bdd5516391f64317b65a8197c179b31e
I20260809 20:28:54.359267 1065 ts_tablet_manager.cc:1939] T bdd5516391f64317b65a8197c179b31e P f8cd065eb45f4275846881b6f937973c: Deleting consensus metadata
I20260809 20:28:54.360889 1066 ts_tablet_manager.cc:1918] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:54.361196 1066 log.cc:1196] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-0/wal/wals/bdd5516391f64317b65a8197c179b31e
I20260809 20:28:54.361469 524 catalog_manager.cc:4841] TS f8cd065eb45f4275846881b6f937973c (127.12.78.2:34537): tablet bdd5516391f64317b65a8197c179b31e (table test-table [id=3680ca5ee786437b87484f3c422a07b8]) successfully deleted
I20260809 20:28:54.361994 1066 ts_tablet_manager.cc:1939] T bdd5516391f64317b65a8197c179b31e P f0fa29b05f2d47f89ff2b5c6e27348bd: Deleting consensus metadata
I20260809 20:28:54.363696 527 catalog_manager.cc:4841] TS f0fa29b05f2d47f89ff2b5c6e27348bd (127.12.78.1:37133): tablet bdd5516391f64317b65a8197c179b31e (table test-table [id=3680ca5ee786437b87484f3c422a07b8]) successfully deleted
I20260809 20:28:54.366693 1067 ts_tablet_manager.cc:1918] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:54.366876 1067 log.cc:1196] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/bdd5516391f64317b65a8197c179b31e
I20260809 20:28:54.367489 1067 ts_tablet_manager.cc:1939] T bdd5516391f64317b65a8197c179b31e P 5fc41541bd6645bba1e2a6b922d443e0: Deleting consensus metadata
I20260809 20:28:54.369333 527 catalog_manager.cc:4841] TS 5fc41541bd6645bba1e2a6b922d443e0 (127.12.78.3:34841): tablet bdd5516391f64317b65a8197c179b31e (table test-table [id=3680ca5ee786437b87484f3c422a07b8]) successfully deleted
I20260809 20:28:54.369833 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.344848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.12.78.1:37133)
0809 20:28:54.345105 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:54.345114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:54.345633 (+ 519us) server_negotiation.cc:407] Connection header received
0809 20:28:54.345844 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:54.345849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:54.345937 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:54.346051 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:54.347641 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.348891 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.350824 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.351507 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.351645 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.362362 (+ 10717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:54.362379 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:54.362398 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:54.362451 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:54.366220 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:54.367473 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:54.367482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:54.367484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:54.367689 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:54.368098 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:54.368104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:54.368108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:54.368491 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:54.368872 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:54.369168 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 20:28:54.369420 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 20:28:54.370749 735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:54.478816 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 603
I20260809 20:28:54.505897 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 746
Aug 09 20:28:54 dist-test-slave-rnrw krb5kdc[493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.2@KRBTEST.COM
I20260809 20:28:54.523698 1038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.506799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54978 (local address 127.12.78.2:34537)
0809 20:28:54.506978 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:54.506983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:54.507101 (+ 118us) server_negotiation.cc:407] Connection header received
0809 20:28:54.507268 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:54.507273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:54.507339 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:54.507425 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:54.509145 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.509877 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.511778 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.512178 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.512274 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.517617 (+ 5343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:54.517634 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:54.517650 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:54.517696 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:54.520768 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:54.521508 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:54.521516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:54.521518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:54.521654 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:54.522046 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:54.522052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:54.522055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:54.522459 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:54.522810 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:54.523108 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 20:28:54.523337 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 20:28:54.524441 877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:54.614180 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 746
I20260809 20:28:54.634650 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 888
Aug 09 20:28:54 dist-test-slave-rnrw krb5kdc[493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.3@KRBTEST.COM
I20260809 20:28:54.652181 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:54.635584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.12.78.3:34841)
0809 20:28:54.635785 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:54.635791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:54.635812 (+ 21us) server_negotiation.cc:407] Connection header received
0809 20:28:54.635980 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:54.635984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:54.636051 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:54.636171 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:54.637587 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.638390 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.639896 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:54.640340 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:54.640511 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:54.645595 (+ 5084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:54.645606 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:54.645622 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:54.645668 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:54.649155 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:54.649883 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:54.649892 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:54.649894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:54.650054 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:54.650446 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:54.650452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:54.650455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:54.650890 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:54.651243 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:54.651531 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 20:28:54.651771 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 20:28:54.653126 1019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:54.776427 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 888
I20260809 20:28:54.802245 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 506
I20260809 20:28:54.803442 570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:54.922125 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 506
2026-08-09T20:28:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4725 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T20:28:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:28:54Z Disabled control of system clock
I20260809 20:28:55.017007 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:42595
--webserver_interface=127.12.78.62
--webserver_port=0
--builtin_ntp_servers=127.12.78.20:44663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:42595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 20:28:55.377967 1076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:55.378289 1076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:55.378376 1076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:55.385872 1076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:28:55.385996 1076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:55.386034 1076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:55.386094 1076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:28:55.386139 1076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:28:55.395638 1076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:42595
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:42595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:55.396982 1076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:55.398767 1076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:55.410983 1083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:55.411062 1085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:55.411065 1082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:55.412079 1076 server_base.cc:1034] running on GCE node
I20260809 20:28:55.413020 1076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:55.414451 1076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:55.415668 1076 hybrid_clock.cc:648] HybridClock initialized: now 1786307335415627 us; error 31 us; skew 500 ppm
I20260809 20:28:55.415982 1076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:28:55.418274 1076 webserver.cc:460] Webserver started at http://127.12.78.62:40515/ using document root <none> and password file <none>
I20260809 20:28:55.418949 1076 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:55.419039 1076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:55.419313 1076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:55.422362 1076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "bba3d0712a024c2dac14b77a0bda0ab7"
format_stamp: "Formatted at 2026-08-09 20:28:55 on dist-test-slave-rnrw"
server_key: "66732de2257c32aa4099abc6f2ddd225"
server_key_iv: "f4b7c2d6ff349fd0f1ce2e30ea5d1712"
server_key_version: "encryptionkey@0"
I20260809 20:28:55.423270 1076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "bba3d0712a024c2dac14b77a0bda0ab7"
format_stamp: "Formatted at 2026-08-09 20:28:55 on dist-test-slave-rnrw"
server_key: "66732de2257c32aa4099abc6f2ddd225"
server_key_iv: "f4b7c2d6ff349fd0f1ce2e30ea5d1712"
server_key_version: "encryptionkey@0"
I20260809 20:28:55.428874 1076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260809 20:28:55.432685 1091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:55.434257 1076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:28:55.434460 1076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "bba3d0712a024c2dac14b77a0bda0ab7"
format_stamp: "Formatted at 2026-08-09 20:28:55 on dist-test-slave-rnrw"
server_key: "66732de2257c32aa4099abc6f2ddd225"
server_key_iv: "f4b7c2d6ff349fd0f1ce2e30ea5d1712"
server_key_version: "encryptionkey@0"
I20260809 20:28:55.434713 1076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:55.447187 1076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:55.448148 1076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:55.448449 1076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:55.468107 1076 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:42595
I20260809 20:28:55.468130 1144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:42595 every 8 connection(s)
I20260809 20:28:55.469570 1076 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:28:55.473377 1145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:55.474011 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 1076
I20260809 20:28:55.474188 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:28:55.474689 12600 external_mini_cluster.cc:1428] Setting key 4c5907c80f5618806ab381ecd8f7f80f
I20260809 20:28:55.489606 1145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7: Bootstrap starting.
I20260809 20:28:55.491688 1148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:55.476623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52234 (local address 127.12.78.62:42595)
0809 20:28:55.477763 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:55.477778 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:55.477829 (+ 51us) server_negotiation.cc:407] Connection header received
0809 20:28:55.480310 (+ 2481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:55.480339 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:55.481027 (+ 688us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:55.481504 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:55.482076 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:55.483958 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:55.485407 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:55.486877 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:55.487246 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:55.488415 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:55.488427 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:55.488450 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:55.489306 (+ 856us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:55.489424 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:55.489457 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:55.489604 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:55.489827 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 20:28:55.490220 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":335,"threads_started":1}
I20260809 20:28:55.493873 1145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:55.495200 1145 log.cc:824] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:55.498330 1145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7: No bootstrap required, opened a new log
I20260809 20:28:55.506418 1145 raft_consensus.cc:348] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba3d0712a024c2dac14b77a0bda0ab7" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42595 } }
I20260809 20:28:55.506944 1145 raft_consensus.cc:374] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:55.507078 1145 raft_consensus.cc:729] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba3d0712a024c2dac14b77a0bda0ab7, State: Initialized, Role: FOLLOWER
I20260809 20:28:55.507763 1145 consensus_queue.cc:260] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba3d0712a024c2dac14b77a0bda0ab7" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42595 } }
I20260809 20:28:55.508013 1145 raft_consensus.cc:388] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:28:55.508152 1145 raft_consensus.cc:482] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:28:55.508323 1145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:55.510663 1145 raft_consensus.cc:504] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba3d0712a024c2dac14b77a0bda0ab7" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42595 } }
I20260809 20:28:55.511215 1145 leader_election.cc:302] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bba3d0712a024c2dac14b77a0bda0ab7; no voters:
I20260809 20:28:55.511884 1145 leader_election.cc:288] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:28:55.512059 1152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:28:55.513236 1152 raft_consensus.cc:686] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [term 1 LEADER]: Becoming Leader. State: Replica: bba3d0712a024c2dac14b77a0bda0ab7, State: Running, Role: LEADER
I20260809 20:28:55.513760 1152 consensus_queue.cc:237] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba3d0712a024c2dac14b77a0bda0ab7" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42595 } }
I20260809 20:28:55.514232 1145 sys_catalog.cc:564] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:28:55.516892 1153 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bba3d0712a024c2dac14b77a0bda0ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba3d0712a024c2dac14b77a0bda0ab7" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42595 } } }
I20260809 20:28:55.517271 1154 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bba3d0712a024c2dac14b77a0bda0ab7. Latest consensus state: current_term: 1 leader_uuid: "bba3d0712a024c2dac14b77a0bda0ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba3d0712a024c2dac14b77a0bda0ab7" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 42595 } } }
I20260809 20:28:55.517544 1153 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:55.517624 1154 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:55.518711 1159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:28:55.524955 1159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:28:55.535327 1159 catalog_manager.cc:1347] Generated new cluster ID: 7f585c55a1c44831b3cd5b247bf99d4e
I20260809 20:28:55.535504 1159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:28:55.565503 1159 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:28:55.566850 1159 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:28:55.575449 1159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7: Generated new TSK 0
I20260809 20:28:55.576692 1159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:28:55.594905 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.1:0
--local_ip_for_outbound_sockets=127.12.78.1
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42595
--builtin_ntp_servers=127.12.78.20:44663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 20:28:55.949653 1172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:55.949935 1172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:55.950032 1172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:55.957713 1172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:55.957825 1172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:55.957934 1172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.1
I20260809 20:28:55.967475 1172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.78.1
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.1
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:55.968905 1172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:55.970749 1172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:55.984525 1177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:55.984754 1178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:55.985359 1180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:55.990255 1172 server_base.cc:1034] running on GCE node
I20260809 20:28:55.991075 1172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:55.992343 1172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:55.993654 1172 hybrid_clock.cc:648] HybridClock initialized: now 1786307335993637 us; error 114 us; skew 500 ppm
I20260809 20:28:55.994045 1172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:28:55.996718 1172 webserver.cc:460] Webserver started at http://127.12.78.1:38279/ using document root <none> and password file <none>
I20260809 20:28:55.997527 1172 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:55.997625 1172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:55.998013 1172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:56.001247 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/data/instance:
uuid: "d2b9ca54539347e1a707a3139c4fbc95"
format_stamp: "Formatted at 2026-08-09 20:28:55 on dist-test-slave-rnrw"
server_key: "15a36a1419446213b9a0774190503e3e"
server_key_iv: "9238a5b56d3ad4aa2b34ea02d4266c48"
server_key_version: "encryptionkey@0"
I20260809 20:28:56.002125 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance:
uuid: "d2b9ca54539347e1a707a3139c4fbc95"
format_stamp: "Formatted at 2026-08-09 20:28:55 on dist-test-slave-rnrw"
server_key: "15a36a1419446213b9a0774190503e3e"
server_key_iv: "9238a5b56d3ad4aa2b34ea02d4266c48"
server_key_version: "encryptionkey@0"
I20260809 20:28:56.007851 1172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:28:56.011791 1187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:56.013370 1172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:28:56.013587 1172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/wal
uuid: "d2b9ca54539347e1a707a3139c4fbc95"
format_stamp: "Formatted at 2026-08-09 20:28:55 on dist-test-slave-rnrw"
server_key: "15a36a1419446213b9a0774190503e3e"
server_key_iv: "9238a5b56d3ad4aa2b34ea02d4266c48"
server_key_version: "encryptionkey@0"
I20260809 20:28:56.013834 1172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:56.034025 1172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:56.034958 1172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:56.035264 1172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:56.036427 1172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:56.037963 1172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:56.038103 1172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:56.038242 1172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:56.038341 1172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:56.072391 1172 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.1:46175
I20260809 20:28:56.072448 1303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.1:46175 every 8 connection(s)
I20260809 20:28:56.073866 1172 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/data/info.pb
I20260809 20:28:56.076531 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 1172
I20260809 20:28:56.076740 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-0/wal/instance
I20260809 20:28:56.077453 12600 external_mini_cluster.cc:1428] Setting key 3f89403e336e4839938a5d6bba7a1414
I20260809 20:28:56.083300 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.2:0
--local_ip_for_outbound_sockets=127.12.78.2
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42595
--builtin_ntp_servers=127.12.78.20:44663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 20:28:56.088369 1307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:56.076727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:52691 (local address 127.12.78.62:42595)
0809 20:28:56.080921 (+ 4194us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:56.080928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:56.080945 (+ 17us) server_negotiation.cc:407] Connection header received
0809 20:28:56.080994 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:56.080999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:56.081109 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:56.081210 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:56.082636 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:56.083458 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:56.085526 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:56.085942 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:56.086082 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:56.087486 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:56.087491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:56.087494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:56.087513 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:56.087554 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:56.087566 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:56.087670 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:56.087896 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 20:28:56.087998 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4066,"thread_start_us":3968,"threads_started":1}
I20260809 20:28:56.089428 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:56.077292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42595 (local address 127.12.78.1:52691)
0809 20:28:56.078393 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:56.078463 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:56.080100 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:56.081519 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:56.081531 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:56.081654 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:56.082443 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:56.082469 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:56.083689 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:56.083697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:56.085372 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:56.085385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:56.086183 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:56.086191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:56.086420 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:56.087143 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:28:56.087146 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:56.087187 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:28:56.087271 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:56.087672 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:56.087687 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:56.087699 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 20:28:56.087884 (+ 185us) client_negotiation.cc:240] Negotiation successful
0809 20:28:56.088304 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":399,"threads_started":1}
I20260809 20:28:56.091174 1305 heartbeater.cc:347] Connected to a master server at 127.12.78.62:42595
I20260809 20:28:56.091559 1305 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:56.092584 1305 heartbeater.cc:511] Master 127.12.78.62:42595 requested a full tablet report, sending...
I20260809 20:28:56.095021 1108 ts_manager.cc:194] Registered new tserver with Master: d2b9ca54539347e1a707a3139c4fbc95 (127.12.78.1:46175)
I20260809 20:28:56.096614 1108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.78.1:52691
W20260809 20:28:56.439173 1309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:56.439533 1309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:56.439708 1309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:56.447114 1309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:56.447261 1309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:56.447454 1309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.2
I20260809 20:28:56.456920 1309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.78.2
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.2
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:56.458256 1309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:56.460049 1309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:56.473206 1318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:56.473316 1316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:56.473824 1315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:56.474284 1309 server_base.cc:1034] running on GCE node
I20260809 20:28:56.474830 1309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:56.475932 1309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:56.477164 1309 hybrid_clock.cc:648] HybridClock initialized: now 1786307336477109 us; error 48 us; skew 500 ppm
I20260809 20:28:56.477504 1309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:28:56.479938 1309 webserver.cc:460] Webserver started at http://127.12.78.2:40689/ using document root <none> and password file <none>
I20260809 20:28:56.480741 1309 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:56.480859 1309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:56.481186 1309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:56.484043 1309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/data/instance:
uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8"
format_stamp: "Formatted at 2026-08-09 20:28:56 on dist-test-slave-rnrw"
server_key: "dad5b7554dfba3608297f4f22ca714d1"
server_key_iv: "ecdde30b78878a2a1cb2b3a357828996"
server_key_version: "encryptionkey@0"
I20260809 20:28:56.484902 1309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance:
uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8"
format_stamp: "Formatted at 2026-08-09 20:28:56 on dist-test-slave-rnrw"
server_key: "dad5b7554dfba3608297f4f22ca714d1"
server_key_iv: "ecdde30b78878a2a1cb2b3a357828996"
server_key_version: "encryptionkey@0"
I20260809 20:28:56.490039 1309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 20:28:56.493552 1324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:56.494889 1309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 20:28:56.495087 1309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/wal
uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8"
format_stamp: "Formatted at 2026-08-09 20:28:56 on dist-test-slave-rnrw"
server_key: "dad5b7554dfba3608297f4f22ca714d1"
server_key_iv: "ecdde30b78878a2a1cb2b3a357828996"
server_key_version: "encryptionkey@0"
I20260809 20:28:56.495321 1309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:56.529917 1309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:56.530828 1309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:56.531124 1309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:56.532198 1309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:56.533775 1309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:56.533891 1309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:56.534029 1309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:56.534126 1309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:56.568368 1309 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.2:37925
I20260809 20:28:56.568470 1439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.2:37925 every 8 connection(s)
I20260809 20:28:56.569756 1309 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/data/info.pb
I20260809 20:28:56.577275 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 1309
I20260809 20:28:56.577479 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/wal/instance
I20260809 20:28:56.577948 12600 external_mini_cluster.cc:1428] Setting key f0ff9d7f67d1894aa8bdded8068d3efb
I20260809 20:28:56.580889 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.78.3:0
--local_ip_for_outbound_sockets=127.12.78.3
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42595
--builtin_ntp_servers=127.12.78.20:44663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 20:28:56.582898 1307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:56.572475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.2:34635 (local address 127.12.78.62:42595)
0809 20:28:56.572695 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:56.572700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:56.574030 (+ 1330us) server_negotiation.cc:407] Connection header received
0809 20:28:56.575884 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:56.575890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:56.575979 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:56.576098 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:56.577352 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:56.578233 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:56.580000 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:56.580421 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:56.580513 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:56.582118 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:56.582123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:56.582126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:56.582144 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:56.582176 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:56.582187 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:56.582258 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:56.582586 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 20:28:56.582670 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 20:28:56.584283 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:56.572976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42595 (local address 127.12.78.2:34635)
0809 20:28:56.573872 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:56.573907 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:56.575584 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:56.576322 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:56.576330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:56.576409 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:56.577142 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:56.577166 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:56.578386 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:56.578391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:56.579841 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:56.579856 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:56.580673 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:56.580684 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:56.580994 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:56.581751 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:28:56.581757 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:56.581811 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:28:56.581912 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:56.582288 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:56.582300 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:56.582311 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 20:28:56.582490 (+ 179us) client_negotiation.cc:240] Negotiation successful
0809 20:28:56.582910 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":286,"threads_started":1}
I20260809 20:28:56.585775 1440 heartbeater.cc:347] Connected to a master server at 127.12.78.62:42595
I20260809 20:28:56.586090 1440 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:56.586966 1440 heartbeater.cc:511] Master 127.12.78.62:42595 requested a full tablet report, sending...
I20260809 20:28:56.588486 1108 ts_manager.cc:194] Registered new tserver with Master: 7cd3202cc46545dfa8ea18a48bb9a0a8 (127.12.78.2:37925)
I20260809 20:28:56.589313 1108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.78.2:34635
W20260809 20:28:56.923638 1443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:56.923920 1443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:56.924011 1443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:56.931264 1443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:56.931366 1443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:56.931473 1443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.78.3
I20260809 20:28:56.940661 1443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.78.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.78.3
--webserver_port=0
--tserver_master_addrs=127.12.78.62:42595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.78.3
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:56.941870 1443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:56.943599 1443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:56.956877 1448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:56.956909 1451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:56.956985 1449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:56.962210 1443 server_base.cc:1034] running on GCE node
I20260809 20:28:56.962822 1443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:56.963774 1443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:56.964980 1443 hybrid_clock.cc:648] HybridClock initialized: now 1786307336964926 us; error 39 us; skew 500 ppm
I20260809 20:28:56.965314 1443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 20:28:56.967654 1443 webserver.cc:460] Webserver started at http://127.12.78.3:45497/ using document root <none> and password file <none>
I20260809 20:28:56.968374 1443 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:56.968497 1443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:56.968976 1443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:56.971858 1443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/data/instance:
uuid: "b5bb40a6829a4ec484c2bc9d5978b51e"
format_stamp: "Formatted at 2026-08-09 20:28:56 on dist-test-slave-rnrw"
server_key: "5ef30b296510c5dfb9c87c92dd5130d1"
server_key_iv: "98c98c7c032c706e0feec3266d149f89"
server_key_version: "encryptionkey@0"
I20260809 20:28:56.972664 1443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance:
uuid: "b5bb40a6829a4ec484c2bc9d5978b51e"
format_stamp: "Formatted at 2026-08-09 20:28:56 on dist-test-slave-rnrw"
server_key: "5ef30b296510c5dfb9c87c92dd5130d1"
server_key_iv: "98c98c7c032c706e0feec3266d149f89"
server_key_version: "encryptionkey@0"
I20260809 20:28:56.977720 1443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260809 20:28:56.981165 1457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:56.982398 1443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:28:56.982568 1443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/wal
uuid: "b5bb40a6829a4ec484c2bc9d5978b51e"
format_stamp: "Formatted at 2026-08-09 20:28:56 on dist-test-slave-rnrw"
server_key: "5ef30b296510c5dfb9c87c92dd5130d1"
server_key_iv: "98c98c7c032c706e0feec3266d149f89"
server_key_version: "encryptionkey@0"
I20260809 20:28:56.982815 1443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:57.009043 1443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:57.009943 1443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:57.010258 1443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:57.011341 1443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 20:28:57.012912 1443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 20:28:57.013024 1443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:57.013165 1443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 20:28:57.013262 1443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:57.050751 1443 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.3:41207
I20260809 20:28:57.050837 1578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.3:41207 every 8 connection(s)
I20260809 20:28:57.052084 1443 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/data/info.pb
I20260809 20:28:57.052706 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 1443
I20260809 20:28:57.052896 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/wal/instance
I20260809 20:28:57.053421 12600 external_mini_cluster.cc:1428] Setting key 74d921034f3aeff593e256b8f77b1afb
I20260809 20:28:57.065490 1307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.054858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.3:45937 (local address 127.12.78.62:42595)
0809 20:28:57.055065 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:57.055072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:57.057239 (+ 2167us) server_negotiation.cc:407] Connection header received
0809 20:28:57.059017 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:57.059022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:57.059108 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:57.059225 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:57.060236 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.061078 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.062611 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.063056 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.063199 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.064626 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:57.064631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:57.064634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:57.064653 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:57.064685 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:57.064697 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:57.064795 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:57.065020 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 20:28:57.065132 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 20:28:57.066488 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.055592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:42595 (local address 127.12.78.3:45937)
0809 20:28:57.057091 (+ 1499us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:57.057134 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:57.058760 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:57.059441 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:57.059450 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:57.059544 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:28:57.060083 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:57.060098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.061220 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.061225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:57.062451 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:57.062485 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.063292 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.063298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:57.063531 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.064240 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 20:28:57.064243 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:28:57.064283 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 20:28:57.064361 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:28:57.064793 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:28:57.064806 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:28:57.064817 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 20:28:57.065007 (+ 190us) client_negotiation.cc:240] Negotiation successful
0809 20:28:57.065404 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":818,"threads_started":1}
I20260809 20:28:57.068058 1579 heartbeater.cc:347] Connected to a master server at 127.12.78.62:42595
I20260809 20:28:57.068413 1579 heartbeater.cc:464] Registering TS with master...
I20260809 20:28:57.069329 1579 heartbeater.cc:511] Master 127.12.78.62:42595 requested a full tablet report, sending...
I20260809 20:28:57.070678 1108 ts_manager.cc:194] Registered new tserver with Master: b5bb40a6829a4ec484c2bc9d5978b51e (127.12.78.3:41207)
I20260809 20:28:57.071441 1108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.78.3:45937
I20260809 20:28:57.081337 12600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 20:28:57.091357 1307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.085478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.12.78.62:42595)
0809 20:28:57.085729 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:57.085736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:57.086096 (+ 360us) server_negotiation.cc:407] Connection header received
0809 20:28:57.086288 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:57.086296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:57.086377 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:57.086462 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:57.086995 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.087818 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.089216 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.089626 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.089730 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.090648 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:57.090659 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:57.090663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:57.090684 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:57.090718 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:57.090730 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:57.090803 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:57.090919 (+ 116us) server_negotiation.cc:299] Negotiation successful
0809 20:28:57.091087 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 20:28:57.099804 1305 heartbeater.cc:503] Master 127.12.78.62:42595 was elected leader, sending a full tablet report...
I20260809 20:28:57.095261 1108 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 20:28:57.102787 1108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 20:28:57.127259 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.114992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.12.78.2:37925)
0809 20:28:57.118570 (+ 3578us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:57.118576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:57.118597 (+ 21us) server_negotiation.cc:407] Connection header received
0809 20:28:57.118674 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:57.118678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:57.119040 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:57.119255 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:57.120057 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.121563 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.124536 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.125979 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.126125 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.126350 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:57.126712 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 20:28:57.126885 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3451,"thread_start_us":3264,"threads_started":1}
I20260809 20:28:57.127275 1591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.114763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:37925 (local address 127.0.0.1:43136)
0809 20:28:57.115929 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:57.115994 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:57.116228 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:57.119236 (+ 3008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:57.119243 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:57.119279 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:57.119848 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:57.119866 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.121709 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.121718 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:57.124289 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:57.124303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.126300 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.126307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:57.126484 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.126528 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 20:28:57.126630 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 20:28:57.126884 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":205,"thread_start_us":1484,"threads_started":1}
I20260809 20:28:57.128111 1594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.115714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.12.78.1:46175)
0809 20:28:57.117013 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:57.117025 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:57.117874 (+ 849us) server_negotiation.cc:407] Connection header received
0809 20:28:57.118243 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:57.118256 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:57.118553 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:57.118878 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:57.119750 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.121485 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.124568 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.125864 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.126034 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.126322 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:57.127572 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0809 20:28:57.127766 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":1339,"threads_started":1}
I20260809 20:28:57.128589 1593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.115394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:46175 (local address 127.0.0.1:58056)
0809 20:28:57.117691 (+ 2297us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:57.117721 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:57.118002 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:57.118869 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:57.118876 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:57.118907 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:57.119488 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:57.119509 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.121686 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.121691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:57.124320 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:57.124334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.126065 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.126071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:57.126836 (+ 765us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.126870 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 20:28:57.126963 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 20:28:57.127100 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2144,"mutex_wait_us":678,"thread_start_us":1993,"threads_started":1}
I20260809 20:28:57.130859 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.119140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:41207 (local address 127.0.0.1:43598)
0809 20:28:57.121710 (+ 2570us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:57.121736 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:57.121859 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:57.123034 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:57.123039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:57.123068 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:57.123494 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:57.123504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.125419 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.125426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:57.127516 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:57.127532 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.130040 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.130046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:57.130248 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.130280 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 20:28:57.130385 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 20:28:57.130522 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":73,"thread_start_us":476,"threads_started":1}
I20260809 20:28:57.130968 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.117568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.12.78.3:41207)
0809 20:28:57.122426 (+ 4858us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:57.122434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:57.122499 (+ 65us) server_negotiation.cc:407] Connection header received
0809 20:28:57.122606 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:57.122614 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:57.122833 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:57.123026 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:57.123722 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.125274 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.127753 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.129835 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.129975 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.130227 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:57.130446 (+ 219us) server_negotiation.cc:299] Negotiation successful
0809 20:28:57.130610 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4672,"thread_start_us":4512,"threads_started":1}
I20260809 20:28:57.131735 1374 tablet_service.cc:1467] Processing CreateTablet for tablet 9e68a599fb864baf84512b71b03a40f9 (DEFAULT_TABLE table=test-table [id=36e2ed2339ca429eb8e14ccc1f29d18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:57.132089 1237 tablet_service.cc:1467] Processing CreateTablet for tablet 9e68a599fb864baf84512b71b03a40f9 (DEFAULT_TABLE table=test-table [id=36e2ed2339ca429eb8e14ccc1f29d18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:57.133248 1374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e68a599fb864baf84512b71b03a40f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:57.133777 1237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e68a599fb864baf84512b71b03a40f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:57.134415 1510 tablet_service.cc:1467] Processing CreateTablet for tablet 9e68a599fb864baf84512b71b03a40f9 (DEFAULT_TABLE table=test-table [id=36e2ed2339ca429eb8e14ccc1f29d18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 20:28:57.135845 1510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e68a599fb864baf84512b71b03a40f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:57.144843 1597 tablet_bootstrap.cc:492] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8: Bootstrap starting.
I20260809 20:28:57.148468 1597 tablet_bootstrap.cc:654] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:57.149561 1597 log.cc:824] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:57.152279 1597 tablet_bootstrap.cc:492] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8: No bootstrap required, opened a new log
I20260809 20:28:57.152632 1597 ts_tablet_manager.cc:1397] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260809 20:28:57.153455 1599 tablet_bootstrap.cc:492] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e: Bootstrap starting.
I20260809 20:28:57.157374 1599 tablet_bootstrap.cc:654] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:57.158666 1599 log.cc:824] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:57.159989 1598 tablet_bootstrap.cc:492] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95: Bootstrap starting.
I20260809 20:28:57.162596 1599 tablet_bootstrap.cc:492] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e: No bootstrap required, opened a new log
I20260809 20:28:57.162870 1599 ts_tablet_manager.cc:1397] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260809 20:28:57.163702 1598 tablet_bootstrap.cc:654] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:57.163869 1597 raft_consensus.cc:348] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37925 } } peers { permanent_uuid: "b5bb40a6829a4ec484c2bc9d5978b51e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41207 } } peers { permanent_uuid: "d2b9ca54539347e1a707a3139c4fbc95" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46175 } }
I20260809 20:28:57.164350 1597 raft_consensus.cc:374] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:57.164462 1597 raft_consensus.cc:729] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cd3202cc46545dfa8ea18a48bb9a0a8, State: Initialized, Role: FOLLOWER
I20260809 20:28:57.165036 1598 log.cc:824] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:57.165158 1597 consensus_queue.cc:260] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37925 } } peers { permanent_uuid: "b5bb40a6829a4ec484c2bc9d5978b51e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41207 } } peers { permanent_uuid: "d2b9ca54539347e1a707a3139c4fbc95" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46175 } }
I20260809 20:28:57.168783 1598 tablet_bootstrap.cc:492] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95: No bootstrap required, opened a new log
I20260809 20:28:57.169153 1598 ts_tablet_manager.cc:1397] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 20:28:57.171510 1599 raft_consensus.cc:348] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37925 } } peers { permanent_uuid: "b5bb40a6829a4ec484c2bc9d5978b51e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41207 } } peers { permanent_uuid: "d2b9ca54539347e1a707a3139c4fbc95" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46175 } }
I20260809 20:28:57.171893 1599 raft_consensus.cc:374] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:57.172014 1599 raft_consensus.cc:729] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5bb40a6829a4ec484c2bc9d5978b51e, State: Initialized, Role: FOLLOWER
I20260809 20:28:57.172345 1440 heartbeater.cc:503] Master 127.12.78.62:42595 was elected leader, sending a full tablet report...
I20260809 20:28:57.172731 1599 consensus_queue.cc:260] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37925 } } peers { permanent_uuid: "b5bb40a6829a4ec484c2bc9d5978b51e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41207 } } peers { permanent_uuid: "d2b9ca54539347e1a707a3139c4fbc95" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46175 } }
I20260809 20:28:57.173549 1597 ts_tablet_manager.cc:1428] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8: Time spent starting tablet: real 0.021s user 0.007s sys 0.012s
I20260809 20:28:57.174433 1579 heartbeater.cc:503] Master 127.12.78.62:42595 was elected leader, sending a full tablet report...
I20260809 20:28:57.175136 1599 ts_tablet_manager.cc:1428] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e: Time spent starting tablet: real 0.012s user 0.003s sys 0.010s
I20260809 20:28:57.180022 1598 raft_consensus.cc:348] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37925 } } peers { permanent_uuid: "b5bb40a6829a4ec484c2bc9d5978b51e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41207 } } peers { permanent_uuid: "d2b9ca54539347e1a707a3139c4fbc95" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46175 } }
I20260809 20:28:57.180477 1598 raft_consensus.cc:374] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:57.180622 1598 raft_consensus.cc:729] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b9ca54539347e1a707a3139c4fbc95, State: Initialized, Role: FOLLOWER
I20260809 20:28:57.181244 1598 consensus_queue.cc:260] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37925 } } peers { permanent_uuid: "b5bb40a6829a4ec484c2bc9d5978b51e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41207 } } peers { permanent_uuid: "d2b9ca54539347e1a707a3139c4fbc95" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46175 } }
I20260809 20:28:57.182659 1598 ts_tablet_manager.cc:1428] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260809 20:28:57.208698 1606 raft_consensus.cc:482] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 20:28:57.209052 1606 raft_consensus.cc:504] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37925 } } peers { permanent_uuid: "b5bb40a6829a4ec484c2bc9d5978b51e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41207 } } peers { permanent_uuid: "d2b9ca54539347e1a707a3139c4fbc95" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46175 } }
I20260809 20:28:57.210772 1606 leader_election.cc:288] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cd3202cc46545dfa8ea18a48bb9a0a8 (127.12.78.2:37925), b5bb40a6829a4ec484c2bc9d5978b51e (127.12.78.3:41207)
I20260809 20:28:57.217720 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.211177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:57893 (local address 127.12.78.2:37925)
0809 20:28:57.211442 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:57.211449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:57.211818 (+ 369us) server_negotiation.cc:407] Connection header received
0809 20:28:57.212101 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:57.212110 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:57.212222 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:57.212324 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:57.212981 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.213685 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.215665 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.216789 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.216911 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.217002 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:57.217277 (+ 275us) server_negotiation.cc:299] Negotiation successful
0809 20:28:57.217412 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 20:28:57.217734 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.211105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.2:37925 (local address 127.12.78.1:57893)
0809 20:28:57.211688 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:57.211713 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:57.211897 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:57.212393 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:57.212398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:57.212424 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:57.212837 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:57.212847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.213814 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.213822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:57.215472 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:57.215484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.216986 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.216991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:57.217120 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.217146 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 20:28:57.217222 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 20:28:57.217312 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"spinlock_wait_cycles":30592,"thread_start_us":299,"threads_started":1}
I20260809 20:28:57.218750 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.212124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.78.1:38639 (local address 127.12.78.3:41207)
0809 20:28:57.212354 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:57.212360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:57.212588 (+ 228us) server_negotiation.cc:407] Connection header received
0809 20:28:57.212808 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:57.212817 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:57.212878 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:57.212976 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 20:28:57.213505 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.214211 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.216340 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.217592 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.217727 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.217849 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:57.218363 (+ 514us) server_negotiation.cc:299] Negotiation successful
0809 20:28:57.218528 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 20:28:57.218763 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.211860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.3:41207 (local address 127.12.78.1:38639)
0809 20:28:57.212464 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:57.212477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 20:28:57.212661 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:28:57.212997 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:28:57.213002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:28:57.213024 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 20:28:57.213351 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:57.213359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.214358 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.214364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:57.216132 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:28:57.216147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.217799 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.217805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:28:57.218149 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.218185 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 20:28:57.218281 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 20:28:57.218418 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":303,"threads_started":1}
I20260809 20:28:57.218958 1394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e68a599fb864baf84512b71b03a40f9" candidate_uuid: "d2b9ca54539347e1a707a3139c4fbc95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8" is_pre_election: true
I20260809 20:28:57.219547 1394 raft_consensus.cc:2454] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2b9ca54539347e1a707a3139c4fbc95 in term 0.
I20260809 20:28:57.219856 1531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e68a599fb864baf84512b71b03a40f9" candidate_uuid: "d2b9ca54539347e1a707a3139c4fbc95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5bb40a6829a4ec484c2bc9d5978b51e" is_pre_election: true
I20260809 20:28:57.220245 1190 leader_election.cc:302] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cd3202cc46545dfa8ea18a48bb9a0a8, d2b9ca54539347e1a707a3139c4fbc95; no voters:
I20260809 20:28:57.220501 1531 raft_consensus.cc:2454] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2b9ca54539347e1a707a3139c4fbc95 in term 0.
I20260809 20:28:57.220726 1606 raft_consensus.cc:2781] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 20:28:57.220901 1606 raft_consensus.cc:482] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 20:28:57.221032 1606 raft_consensus.cc:3037] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:57.223294 1606 raft_consensus.cc:504] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37925 } } peers { permanent_uuid: "b5bb40a6829a4ec484c2bc9d5978b51e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41207 } } peers { permanent_uuid: "d2b9ca54539347e1a707a3139c4fbc95" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46175 } }
I20260809 20:28:57.223917 1606 leader_election.cc:288] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [CANDIDATE]: Term 1 election: Requested vote from peers 7cd3202cc46545dfa8ea18a48bb9a0a8 (127.12.78.2:37925), b5bb40a6829a4ec484c2bc9d5978b51e (127.12.78.3:41207)
I20260809 20:28:57.224537 1394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e68a599fb864baf84512b71b03a40f9" candidate_uuid: "d2b9ca54539347e1a707a3139c4fbc95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8"
I20260809 20:28:57.224807 1394 raft_consensus.cc:3037] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:57.224783 1531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e68a599fb864baf84512b71b03a40f9" candidate_uuid: "d2b9ca54539347e1a707a3139c4fbc95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5bb40a6829a4ec484c2bc9d5978b51e"
I20260809 20:28:57.225006 1531 raft_consensus.cc:3037] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:57.227216 1394 raft_consensus.cc:2454] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2b9ca54539347e1a707a3139c4fbc95 in term 1.
I20260809 20:28:57.227284 1531 raft_consensus.cc:2454] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2b9ca54539347e1a707a3139c4fbc95 in term 1.
I20260809 20:28:57.227756 1190 leader_election.cc:302] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cd3202cc46545dfa8ea18a48bb9a0a8, d2b9ca54539347e1a707a3139c4fbc95; no voters:
I20260809 20:28:57.228099 1606 raft_consensus.cc:2781] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:28:57.228621 1606 raft_consensus.cc:686] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [term 1 LEADER]: Becoming Leader. State: Replica: d2b9ca54539347e1a707a3139c4fbc95, State: Running, Role: LEADER
I20260809 20:28:57.229156 1606 consensus_queue.cc:237] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37925 } } peers { permanent_uuid: "b5bb40a6829a4ec484c2bc9d5978b51e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41207 } } peers { permanent_uuid: "d2b9ca54539347e1a707a3139c4fbc95" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46175 } }
I20260809 20:28:57.234524 1108 catalog_manager.cc:5495] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2b9ca54539347e1a707a3139c4fbc95 (127.12.78.1). New cstate: current_term: 1 leader_uuid: "d2b9ca54539347e1a707a3139c4fbc95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5bb40a6829a4ec484c2bc9d5978b51e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2b9ca54539347e1a707a3139c4fbc95" member_type: VOTER last_known_addr { host: "127.12.78.1" port: 46175 } health_report { overall_health: HEALTHY } } }
I20260809 20:28:57.252981 1594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.247349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.12.78.1:46175)
0809 20:28:57.247614 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:57.247619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:57.247636 (+ 17us) server_negotiation.cc:407] Connection header received
0809 20:28:57.247725 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:57.247728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:57.247814 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:57.247941 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:28:57.248275 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.249085 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.250542 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.250949 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.251071 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.251756 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:28:57.252274 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:28:57.252391 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:57.252625 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 20:28:57.252693 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 20:28:57.261533 1394 raft_consensus.cc:1261] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8 [term 1 FOLLOWER]: Refusing update from remote peer d2b9ca54539347e1a707a3139c4fbc95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:28:57.261534 1531 raft_consensus.cc:1261] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e [term 1 FOLLOWER]: Refusing update from remote peer d2b9ca54539347e1a707a3139c4fbc95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:28:57.262485 1606 consensus_queue.cc:1035] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5bb40a6829a4ec484c2bc9d5978b51e" member_type: VOTER last_known_addr { host: "127.12.78.3" port: 41207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:28:57.262825 1609 consensus_queue.cc:1035] T 9e68a599fb864baf84512b71b03a40f9 P d2b9ca54539347e1a707a3139c4fbc95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cd3202cc46545dfa8ea18a48bb9a0a8" member_type: VOTER last_known_addr { host: "127.12.78.2" port: 37925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:28:57.277050 1615 mvcc.cc:205] Tried to move back new op lower bound from 7316714853412163584 to 7316714853294678016. Current Snapshot: MvccSnapshot[applied={T|T < 7316714853412163584}]
I20260809 20:28:57.290787 1108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:28:57.291056 1108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 20:28:57.294894 1108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bba3d0712a024c2dac14b77a0bda0ab7: Sending DeleteTablet for 3 replicas of tablet 9e68a599fb864baf84512b71b03a40f9
I20260809 20:28:57.296193 1237 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e68a599fb864baf84512b71b03a40f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:57 UTC) from {username='slave'} at 127.0.0.1:58056
I20260809 20:28:57.296285 1374 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e68a599fb864baf84512b71b03a40f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:57 UTC) from {username='slave'} at 127.0.0.1:43136
I20260809 20:28:57.297139 1510 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e68a599fb864baf84512b71b03a40f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 20:28:57 UTC) from {username='slave'} at 127.0.0.1:43598
I20260809 20:28:57.297222 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 1172
I20260809 20:28:57.321696 1594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.299696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.12.78.1:46175)
0809 20:28:57.299932 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:57.299942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:57.309735 (+ 9793us) server_negotiation.cc:407] Connection header received
0809 20:28:57.310313 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:57.310323 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:57.310463 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:57.310591 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:57.311485 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.313194 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.315876 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.316924 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.317064 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.319208 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:57.319223 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:57.319250 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:57.320361 (+ 1111us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:57.320435 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:57.320472 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:57.320650 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:57.320967 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 20:28:57.321117 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 20:28:57.323314 1298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:57.323730 1626 tablet_replica.cc:330] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8: stopping tablet replica
I20260809 20:28:57.324291 1626 raft_consensus.cc:2229] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:28:57.324983 1626 raft_consensus.cc:2258] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:57.325803 1625 tablet_replica.cc:330] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e: stopping tablet replica
I20260809 20:28:57.326228 1625 raft_consensus.cc:2229] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 20:28:57.326748 1625 raft_consensus.cc:2258] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 20:28:57.328714 1626 ts_tablet_manager.cc:1905] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:57.330804 1625 ts_tablet_manager.cc:1905] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 20:28:57.335471 1625 ts_tablet_manager.cc:1918] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:57.335659 1625 log.cc:1196] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-2/wal/wals/9e68a599fb864baf84512b71b03a40f9
I20260809 20:28:57.336144 1625 ts_tablet_manager.cc:1939] T 9e68a599fb864baf84512b71b03a40f9 P b5bb40a6829a4ec484c2bc9d5978b51e: Deleting consensus metadata
I20260809 20:28:57.336274 1626 ts_tablet_manager.cc:1918] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 20:28:57.336444 1626 log.cc:1196] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8: Deleting WAL directory at /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786307001768029-12600-0/minicluster-data/ts-1/wal/wals/9e68a599fb864baf84512b71b03a40f9
I20260809 20:28:57.337049 1626 ts_tablet_manager.cc:1939] T 9e68a599fb864baf84512b71b03a40f9 P 7cd3202cc46545dfa8ea18a48bb9a0a8: Deleting consensus metadata
I20260809 20:28:57.337990 1094 catalog_manager.cc:4841] TS b5bb40a6829a4ec484c2bc9d5978b51e (127.12.78.3:41207): tablet 9e68a599fb864baf84512b71b03a40f9 (table test-table [id=36e2ed2339ca429eb8e14ccc1f29d18b]) successfully deleted
I20260809 20:28:57.338806 1094 catalog_manager.cc:4841] TS 7cd3202cc46545dfa8ea18a48bb9a0a8 (127.12.78.2:37925): tablet 9e68a599fb864baf84512b71b03a40f9 (table test-table [id=36e2ed2339ca429eb8e14ccc1f29d18b]) successfully deleted
I20260809 20:28:57.431284 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 1172
I20260809 20:28:57.452879 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 1309
W20260809 20:28:57.452864 1095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.78.1:46175 (error 108)
I20260809 20:28:57.454773 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.454341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:46175 (local address 127.0.0.1:58080)
0809 20:28:57.454504 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:57.454625 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 20:28:57.455109 1095 catalog_manager.cc:4584] TS d2b9ca54539347e1a707a3139c4fbc95 (127.12.78.1:46175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e68a599fb864baf84512b71b03a40f9: Network error: Client connection negotiation failed: client connection to 127.12.78.1:46175: connect: Connection refused (error 111)
I20260809 20:28:57.460201 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.453775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.12.78.2:37925)
0809 20:28:57.453997 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:57.454003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:57.454027 (+ 24us) server_negotiation.cc:407] Connection header received
0809 20:28:57.454263 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:57.454269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:57.454372 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:57.454501 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:57.454924 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.455756 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.457160 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.457559 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.457692 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.458839 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:57.458848 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:57.458861 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:57.459440 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:57.459485 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:57.459504 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:57.459629 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:57.459771 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 20:28:57.459880 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 20:28:57.460935 1435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:57.508805 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.508227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:46175 (local address 127.0.0.1:58090)
0809 20:28:57.508469 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:57.508619 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 20:28:57.509945 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.509608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:46175 (local address 127.0.0.1:58092)
0809 20:28:57.509743 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:57.509814 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 20:28:57.550855 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 1309
I20260809 20:28:57.570708 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 1443
I20260809 20:28:57.570830 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.570338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:46175 (local address 127.0.0.1:58106)
0809 20:28:57.570564 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:57.570659 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 20:28:57.572100 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.571657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:46175 (local address 127.0.0.1:58108)
0809 20:28:57.571836 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:57.571942 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 20:28:57.577690 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.571536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.12.78.3:41207)
0809 20:28:57.571733 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:57.571739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:57.571763 (+ 24us) server_negotiation.cc:407] Connection header received
0809 20:28:57.571955 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:57.571961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:57.572086 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:57.572221 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:57.572651 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.573411 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.574732 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:57.575123 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:57.575217 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:57.576276 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:57.576283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:57.576297 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:28:57.577074 (+ 777us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:57.577122 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:28:57.577143 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:57.577238 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:57.577370 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 20:28:57.577460 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 20:28:57.578389 1573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:57.652860 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.652327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:46175 (local address 127.0.0.1:58120)
0809 20:28:57.652538 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:57.652703 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 20:28:57.653957 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:57.653546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.1:46175 (local address 127.0.0.1:58122)
0809 20:28:57.653772 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 20:28:57.653840 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.1:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260809 20:28:57.668248 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 1443
I20260809 20:28:57.687768 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 1076
I20260809 20:28:57.688967 1140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:28:57.799015 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 1076
2026-08-09T20:28:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2878 ms)
[----------] 6 tests from EncryptionPolicyTest (23139 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:28:57 dist-test-slave-rnrw krb5kdc[1636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:28:57 dist-test-slave-rnrw krb5kdc[1636](info): set up 2 sockets
Aug 09 20:28:57 dist-test-slave-rnrw krb5kdc[1636](info): commencing operation
krb5kdc: starting...
W20260809 20:28:59.016341 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.004s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:28:59 dist-test-slave-rnrw krb5kdc[1636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:28:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:28:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 20:28:59.336047 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:28:59.694967 1651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:28:59.695291 1651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:28:59.695464 1651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:28:59.702979 1651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:28:59.703104 1651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:28:59.703204 1651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:28:59.703315 1651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:28:59.703384 1651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:28:59.714066 1651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:28:59.715481 1651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:28:59.716933 1651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:28:59.729185 1656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:59.729154 1659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:28:59.729105 1657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:28:59.730067 1651 server_base.cc:1034] running on GCE node
I20260809 20:28:59.730916 1651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:28:59.732431 1651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:28:59.733668 1651 hybrid_clock.cc:648] HybridClock initialized: now 1786307339733627 us; error 47 us; skew 500 ppm
I20260809 20:28:59.734035 1651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:28:59 dist-test-slave-rnrw krb5kdc[1636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:28:59.737419 1651 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 20:28:59.739012 1651 webserver.cc:460] Webserver started at http://127.0.0.1:35523/ using document root <none> and password file <none>
I20260809 20:28:59.739742 1651 fs_manager.cc:359] Metadata directory not provided
I20260809 20:28:59.739902 1651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:28:59.740226 1651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:28:59.743381 1651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "2f013f0c498b4845802ba55b8a227809"
format_stamp: "Formatted at 2026-08-09 20:28:59 on dist-test-slave-rnrw"
server_key: "ac1d13a01718d6520481779f732df582"
server_key_iv: "e3b4b2439462af37bce17e84f439ac62"
server_key_version: "encryptionkey@0"
I20260809 20:28:59.744215 1651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "2f013f0c498b4845802ba55b8a227809"
format_stamp: "Formatted at 2026-08-09 20:28:59 on dist-test-slave-rnrw"
server_key: "ac1d13a01718d6520481779f732df582"
server_key_iv: "e3b4b2439462af37bce17e84f439ac62"
server_key_version: "encryptionkey@0"
I20260809 20:28:59.749907 1651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 20:28:59.753690 1666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:28:59.755013 1651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 20:28:59.755218 1651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "2f013f0c498b4845802ba55b8a227809"
format_stamp: "Formatted at 2026-08-09 20:28:59 on dist-test-slave-rnrw"
server_key: "ac1d13a01718d6520481779f732df582"
server_key_iv: "e3b4b2439462af37bce17e84f439ac62"
server_key_version: "encryptionkey@0"
I20260809 20:28:59.755432 1651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:28:59.788645 1651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:28:59.793043 1651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:28:59.793345 1651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:28:59.813066 1651 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34547
I20260809 20:28:59.813156 1719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34547 every 8 connection(s)
I20260809 20:28:59.814805 1651 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:28:59.818723 1720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:28:59.819049 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 1651
I20260809 20:28:59.819280 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:28:59.820135 12600 external_mini_cluster.cc:1428] Setting key 8637398a3d32fc782eab5db55907dfa8
I20260809 20:28:59.831775 1720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809: Bootstrap starting.
Aug 09 20:28:59 dist-test-slave-rnrw krb5kdc[1636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 20:28:59.835155 1720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809: Neither blocks nor log segments found. Creating new log.
I20260809 20:28:59.836257 1720 log.cc:824] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809: Log is configured to *not* fsync() on all Append() calls
I20260809 20:28:59.839711 1720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809: No bootstrap required, opened a new log
I20260809 20:28:59.844048 1723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:59.822259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60949 (local address 127.0.0.1:34547)
0809 20:28:59.823028 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:59.823040 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:59.823093 (+ 53us) server_negotiation.cc:407] Connection header received
0809 20:28:59.825402 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:59.825425 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:59.825843 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:59.826286 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:59.827828 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:59.829214 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:59.830551 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:59.831253 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:59.831467 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:59.835920 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:59.835929 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:59.835943 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:59.835991 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:59.839582 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:59.840456 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:59.840463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:59.840469 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:59.840707 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:59.841194 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:59.841199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:59.841206 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:59.841779 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:59.842149 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:59.842662 (+ 513us) server_negotiation.cc:299] Negotiation successful
0809 20:28:59.843027 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":346,"threads_started":1}
I20260809 20:28:59.848793 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:28:59.848845 1720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f013f0c498b4845802ba55b8a227809" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } }
I20260809 20:28:59.849292 1720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:28:59.849411 1720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f013f0c498b4845802ba55b8a227809, State: Initialized, Role: FOLLOWER
I20260809 20:28:59.849973 1720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f013f0c498b4845802ba55b8a227809" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } }
I20260809 20:28:59.850234 1720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:28:59.850379 1720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:28:59.850553 1720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:28:59.853050 1720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f013f0c498b4845802ba55b8a227809" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } }
I20260809 20:28:59.853571 1720 leader_election.cc:302] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f013f0c498b4845802ba55b8a227809; no voters:
I20260809 20:28:59.854311 1720 leader_election.cc:288] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:28:59.854699 1729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:28:59.855351 1729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [term 1 LEADER]: Becoming Leader. State: Replica: 2f013f0c498b4845802ba55b8a227809, State: Running, Role: LEADER
I20260809 20:28:59.855979 1729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f013f0c498b4845802ba55b8a227809" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } }
I20260809 20:28:59.858937 1720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:28:59.859248 1732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f013f0c498b4845802ba55b8a227809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f013f0c498b4845802ba55b8a227809" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } } }
I20260809 20:28:59.859645 1732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:59.860013 1731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f013f0c498b4845802ba55b8a227809. Latest consensus state: current_term: 1 leader_uuid: "2f013f0c498b4845802ba55b8a227809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f013f0c498b4845802ba55b8a227809" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } } }
I20260809 20:28:59.860304 1731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809 [sys.catalog]: This master's current role is: LEADER
I20260809 20:28:59.878432 1723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:28:59.854233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44621 (local address 127.0.0.1:34547)
0809 20:28:59.854446 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 20:28:59.854454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:28:59.855220 (+ 766us) server_negotiation.cc:407] Connection header received
0809 20:28:59.855670 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:28:59.855679 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:28:59.855788 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:28:59.855884 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:28:59.858423 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:59.859904 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:59.862474 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:28:59.863448 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:28:59.863575 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:28:59.868110 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:28:59.868120 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:28:59.868128 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:28:59.868201 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:28:59.873230 (+ 5029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:59.874799 (+ 1569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:59.874808 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:59.874815 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:59.875020 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:28:59.875802 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:28:59.875810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:28:59.875816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:28:59.876410 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:28:59.876790 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0809 20:28:59.877542 (+ 752us) server_negotiation.cc:299] Negotiation successful
0809 20:28:59.877821 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 20:28:59.878960 1735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:28:59.884389 1735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:28:59.899336 1735 catalog_manager.cc:1347] Generated new cluster ID: 04f9ae616ca64fc3b7a50fd75fd45b85
I20260809 20:28:59.899480 1735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:28:59.935771 1735 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:28:59.937296 1735 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:28:59.945636 1735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f013f0c498b4845802ba55b8a227809: Generated new TSK 0
I20260809 20:28:59.946542 1735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:28:59.956316 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 1651
I20260809 20:28:59.958209 1715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:29:00.052421 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 1651
W20260809 20:29:00.072981 1633 connection.cc:535] client connection to 127.0.0.1:34547 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34547): Connection reset by peer (error 104)
2026-08-09T20:29:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:00 dist-test-slave-rnrw krb5kdc[1753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:00 dist-test-slave-rnrw krb5kdc[1753](info): set up 2 sockets
Aug 09 20:29:00 dist-test-slave-rnrw krb5kdc[1753](info): commencing operation
krb5kdc: starting...
W20260809 20:29:01.221216 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:01 dist-test-slave-rnrw krb5kdc[1753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 20:29:01.532320 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:01.875676 1766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:01.875977 1766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:01.876175 1766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:01.883473 1766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:01.883584 1766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:01.883703 1766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:01.883811 1766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:01.883891 1766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:01.894115 1766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:01.895491 1766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:01.896884 1766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:01.908968 1771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:01.908973 1772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:01.908979 1774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:01.910151 1766 server_base.cc:1034] running on GCE node
I20260809 20:29:01.911033 1766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:01.912425 1766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:01.913643 1766 hybrid_clock.cc:648] HybridClock initialized: now 1786307341913600 us; error 31 us; skew 500 ppm
I20260809 20:29:01.914013 1766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:01 dist-test-slave-rnrw krb5kdc[1753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:01.917356 1766 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 20:29:01.918984 1766 webserver.cc:460] Webserver started at http://127.0.0.1:36141/ using document root <none> and password file <none>
I20260809 20:29:01.919694 1766 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:01.919847 1766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:01.920161 1766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:01.923372 1766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "8d985c34b553411e9e296033c9a3b16b"
format_stamp: "Formatted at 2026-08-09 20:29:01 on dist-test-slave-rnrw"
server_key: "82f0e16c9c57f7647fc7613fc2e6d201"
server_key_iv: "6ae7109fff0dc7e134cd170cb4e532f6"
server_key_version: "encryptionkey@0"
I20260809 20:29:01.924182 1766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "8d985c34b553411e9e296033c9a3b16b"
format_stamp: "Formatted at 2026-08-09 20:29:01 on dist-test-slave-rnrw"
server_key: "82f0e16c9c57f7647fc7613fc2e6d201"
server_key_iv: "6ae7109fff0dc7e134cd170cb4e532f6"
server_key_version: "encryptionkey@0"
I20260809 20:29:01.929550 1766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 20:29:01.933275 1781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:01.934748 1766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:29:01.934943 1766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "8d985c34b553411e9e296033c9a3b16b"
format_stamp: "Formatted at 2026-08-09 20:29:01 on dist-test-slave-rnrw"
server_key: "82f0e16c9c57f7647fc7613fc2e6d201"
server_key_iv: "6ae7109fff0dc7e134cd170cb4e532f6"
server_key_version: "encryptionkey@0"
I20260809 20:29:01.935154 1766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:01.953469 1766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:01.957967 1766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:01.958271 1766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:01.978073 1766 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46513
I20260809 20:29:01.978116 1835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46513 every 8 connection(s)
I20260809 20:29:01.979526 1766 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:01.981302 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 1766
I20260809 20:29:01.981495 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:01.982082 12600 external_mini_cluster.cc:1428] Setting key a8dacb46b67ddd4e55ed4b15e8ccf82b
I20260809 20:29:01.984331 1836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:01.995731 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:01.983910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60159 (local address 127.0.0.1:46513)
0809 20:29:01.984767 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:01.984779 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:01.984822 (+ 43us) server_negotiation.cc:407] Connection header received
0809 20:29:01.986449 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:01.986471 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:01.986866 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:01.987226 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:01.987876 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:01.989308 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:01.991184 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:01.991838 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:01.991970 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:01.993343 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:01.993354 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:01.993367 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:01.993989 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:01.994070 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:01.994106 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:01.994237 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:01.994510 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 20:29:01.994729 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":402,"threads_started":1}
I20260809 20:29:01.997326 1836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b: Bootstrap starting.
I20260809 20:29:01.999787 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:02.000824 1836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:02.001850 1836 log.cc:824] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:02.004925 1836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b: No bootstrap required, opened a new log
I20260809 20:29:02.011773 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:02.004513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46221 (local address 127.0.0.1:46513)
0809 20:29:02.004716 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:02.004722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:02.005288 (+ 566us) server_negotiation.cc:407] Connection header received
0809 20:29:02.005500 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:02.005503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:02.005564 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:02.005655 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:02.006282 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:02.007061 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:02.009109 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:02.009574 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:02.009673 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:02.010970 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:02.010975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:02.010978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:02.010996 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:02.011033 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:02.011047 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:02.011146 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:02.011336 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 20:29:02.011384 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 20:29:02.014760 1836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d985c34b553411e9e296033c9a3b16b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46513 } }
I20260809 20:29:02.015167 1836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:02.015286 1836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d985c34b553411e9e296033c9a3b16b, State: Initialized, Role: FOLLOWER
I20260809 20:29:02.015856 1836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d985c34b553411e9e296033c9a3b16b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46513 } }
I20260809 20:29:02.016101 1836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:02.016247 1836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:02.016422 1836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:02.019038 1836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d985c34b553411e9e296033c9a3b16b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46513 } }
I20260809 20:29:02.019539 1836 leader_election.cc:302] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d985c34b553411e9e296033c9a3b16b; no voters:
I20260809 20:29:02.020141 1836 leader_election.cc:288] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:02.020298 1847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:02.021081 1847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [term 1 LEADER]: Becoming Leader. State: Replica: 8d985c34b553411e9e296033c9a3b16b, State: Running, Role: LEADER
I20260809 20:29:02.021687 1847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d985c34b553411e9e296033c9a3b16b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46513 } }
I20260809 20:29:02.022271 1836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:02.024852 1848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d985c34b553411e9e296033c9a3b16b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d985c34b553411e9e296033c9a3b16b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46513 } } }
I20260809 20:29:02.025152 1849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d985c34b553411e9e296033c9a3b16b. Latest consensus state: current_term: 1 leader_uuid: "8d985c34b553411e9e296033c9a3b16b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d985c34b553411e9e296033c9a3b16b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46513 } } }
I20260809 20:29:02.025674 1848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:02.025749 1849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:02.029326 1852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:02.037169 1852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:02.059296 1852 catalog_manager.cc:1347] Generated new cluster ID: 83d8724e6b9e41efad24f9b95926cc37
I20260809 20:29:02.059427 1852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:02.081022 1852 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:02.082345 1852 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:02.098732 1852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d985c34b553411e9e296033c9a3b16b: Generated new TSK 0
I20260809 20:29:02.099610 1852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:02.109614 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 1766
W20260809 20:29:02.110996 1831 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60159
W20260809 20:29:02.111979 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 1766: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:29:02.112182 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 1766
W20260809 20:29:02.133270 1751 connection.cc:535] client connection to 127.0.0.1:46513 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46513): Connection reset by peer (error 104)
2026-08-09T20:29:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2060 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:02 dist-test-slave-rnrw krb5kdc[1868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:02 dist-test-slave-rnrw krb5kdc[1868](info): set up 2 sockets
Aug 09 20:29:02 dist-test-slave-rnrw krb5kdc[1868](info): commencing operation
krb5kdc: starting...
W20260809 20:29:03.296069 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:03 dist-test-slave-rnrw krb5kdc[1868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 20:29:03.605958 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:03.957368 1882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:03.957685 1882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:03.957780 1882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:03.965093 1882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:03.965200 1882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:03.965246 1882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:03.965306 1882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:03.965350 1882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:03.975484 1882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:03.976814 1882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:03.978118 1882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:03.990398 1890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:03.990489 1887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:03.990473 1888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:03.991688 1882 server_base.cc:1034] running on GCE node
I20260809 20:29:03.992702 1882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:03.994333 1882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:03.995585 1882 hybrid_clock.cc:648] HybridClock initialized: now 1786307343995542 us; error 37 us; skew 500 ppm
I20260809 20:29:03.995951 1882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:03 dist-test-slave-rnrw krb5kdc[1868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:03.999383 1882 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 20:29:04.001150 1882 webserver.cc:460] Webserver started at http://127.0.0.1:44537/ using document root <none> and password file <none>
I20260809 20:29:04.001870 1882 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:04.002023 1882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:04.002370 1882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:04.005729 1882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "eba60ae0659446bf8e9e474663897490"
format_stamp: "Formatted at 2026-08-09 20:29:03 on dist-test-slave-rnrw"
server_key: "72797eca6f4822201959cd9bfcaf3ff9"
server_key_iv: "2c999822dafadc97d3e0c680569bdd22"
server_key_version: "encryptionkey@0"
I20260809 20:29:04.006592 1882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "eba60ae0659446bf8e9e474663897490"
format_stamp: "Formatted at 2026-08-09 20:29:03 on dist-test-slave-rnrw"
server_key: "72797eca6f4822201959cd9bfcaf3ff9"
server_key_iv: "2c999822dafadc97d3e0c680569bdd22"
server_key_version: "encryptionkey@0"
I20260809 20:29:04.012248 1882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 20:29:04.015980 1897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:04.017457 1882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:29:04.017658 1882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "eba60ae0659446bf8e9e474663897490"
format_stamp: "Formatted at 2026-08-09 20:29:03 on dist-test-slave-rnrw"
server_key: "72797eca6f4822201959cd9bfcaf3ff9"
server_key_iv: "2c999822dafadc97d3e0c680569bdd22"
server_key_version: "encryptionkey@0"
I20260809 20:29:04.017871 1882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:04.023955 1882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:04.028599 1882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:04.028929 1882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:04.048329 1882 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34757
I20260809 20:29:04.048377 1951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34757 every 8 connection(s)
I20260809 20:29:04.049887 1882 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:04.053803 1952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:04.056231 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 1882
I20260809 20:29:04.056425 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:04.057019 12600 external_mini_cluster.cc:1428] Setting key 585354e04562080a3373e7b1d68515d3
I20260809 20:29:04.065202 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:04.058979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37707 (local address 127.0.0.1:34757)
0809 20:29:04.059876 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:04.059887 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:04.059933 (+ 46us) server_negotiation.cc:407] Connection header received
0809 20:29:04.061743 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:04.061768 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:04.062115 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:04.062455 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:04.063533 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:04.063539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:04.063551 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:04.063568 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:04.063627 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:04.063650 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:04.063829 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:04.063936 (+ 107us) server_negotiation.cc:299] Negotiation successful
0809 20:29:04.064173 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":424,"threads_started":1}
I20260809 20:29:04.066416 1952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490: Bootstrap starting.
I20260809 20:29:04.069116 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:04.069967 1952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:04.071230 1952 log.cc:824] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:04.074122 1952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490: No bootstrap required, opened a new log
I20260809 20:29:04.077026 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:04.074440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35465 (local address 127.0.0.1:34757)
0809 20:29:04.074647 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:04.074656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:04.075023 (+ 367us) server_negotiation.cc:407] Connection header received
0809 20:29:04.075161 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:04.075166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:04.075223 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:04.075336 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:04.076279 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:04.076285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:04.076290 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:04.076310 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:04.076360 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:04.076385 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:04.076524 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:04.076660 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 20:29:04.076721 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 20:29:04.082806 1952 raft_consensus.cc:348] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba60ae0659446bf8e9e474663897490" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34757 } }
I20260809 20:29:04.083191 1952 raft_consensus.cc:374] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:04.083271 1952 raft_consensus.cc:729] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba60ae0659446bf8e9e474663897490, State: Initialized, Role: FOLLOWER
I20260809 20:29:04.083856 1952 consensus_queue.cc:260] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba60ae0659446bf8e9e474663897490" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34757 } }
I20260809 20:29:04.084105 1952 raft_consensus.cc:388] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:04.084295 1952 raft_consensus.cc:482] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:04.084491 1952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:04.086849 1952 raft_consensus.cc:504] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba60ae0659446bf8e9e474663897490" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34757 } }
I20260809 20:29:04.087396 1952 leader_election.cc:302] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eba60ae0659446bf8e9e474663897490; no voters:
I20260809 20:29:04.088107 1952 leader_election.cc:288] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:04.088240 1964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:04.089020 1964 raft_consensus.cc:686] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [term 1 LEADER]: Becoming Leader. State: Replica: eba60ae0659446bf8e9e474663897490, State: Running, Role: LEADER
I20260809 20:29:04.089627 1964 consensus_queue.cc:237] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba60ae0659446bf8e9e474663897490" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34757 } }
I20260809 20:29:04.090322 1952 sys_catalog.cc:564] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:04.092685 1965 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eba60ae0659446bf8e9e474663897490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba60ae0659446bf8e9e474663897490" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34757 } } }
I20260809 20:29:04.092713 1966 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba60ae0659446bf8e9e474663897490. Latest consensus state: current_term: 1 leader_uuid: "eba60ae0659446bf8e9e474663897490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba60ae0659446bf8e9e474663897490" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34757 } } }
I20260809 20:29:04.093221 1966 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:04.093221 1965 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:04.097146 1971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:04.102478 1971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:04.111831 1971 catalog_manager.cc:1347] Generated new cluster ID: 588dced27ff84542812dd9abcea12ccd
I20260809 20:29:04.111961 1971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:04.120352 1971 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:04.120899 1971 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:04.139508 1971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eba60ae0659446bf8e9e474663897490: Generated new TSK 0
I20260809 20:29:04.140663 1971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:04.146116 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 1882
W20260809 20:29:04.147238 1947 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37707
W20260809 20:29:04.148029 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 1882: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:29:04.148200 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 1882
2026-08-09T20:29:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2034 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:04 dist-test-slave-rnrw krb5kdc[1986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:04 dist-test-slave-rnrw krb5kdc[1986](info): set up 2 sockets
Aug 09 20:29:04 dist-test-slave-rnrw krb5kdc[1986](info): commencing operation
krb5kdc: starting...
W20260809 20:29:05.312196 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:05 dist-test-slave-rnrw krb5kdc[1986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 20:29:05.624176 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42933
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:05.977087 2001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:05.977397 2001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:05.977598 2001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:05.985260 2001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:05.985390 2001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:05.985505 2001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:05.985602 2001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:05.985688 2001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:05.996119 2001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:05.997541 2001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:05.998946 2001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:06.011703 2007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:06.011850 2009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:06.011850 2006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:06.012912 2001 server_base.cc:1034] running on GCE node
I20260809 20:29:06.013744 2001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:06.015278 2001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:06.016485 2001 hybrid_clock.cc:648] HybridClock initialized: now 1786307346016429 us; error 46 us; skew 500 ppm
I20260809 20:29:06.016889 2001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:06 dist-test-slave-rnrw krb5kdc[1986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:06.024521 2001 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 20:29:06.026350 2001 webserver.cc:460] Webserver started at http://127.0.0.1:34381/ using document root <none> and password file <none>
I20260809 20:29:06.027125 2001 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:06.027289 2001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:06.027659 2001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:06.031066 2001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "e5ab782aa1a6446d9b3bfed988be7906"
format_stamp: "Formatted at 2026-08-09 20:29:06 on dist-test-slave-rnrw"
server_key: "9c499f364d78a0ee397fe675031541e9"
server_key_iv: "69e94f4b60aec929389e111309564061"
server_key_version: "encryptionkey@0"
I20260809 20:29:06.031955 2001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "e5ab782aa1a6446d9b3bfed988be7906"
format_stamp: "Formatted at 2026-08-09 20:29:06 on dist-test-slave-rnrw"
server_key: "9c499f364d78a0ee397fe675031541e9"
server_key_iv: "69e94f4b60aec929389e111309564061"
server_key_version: "encryptionkey@0"
I20260809 20:29:06.037639 2001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 20:29:06.041545 2018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:06.042989 2001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:29:06.043206 2001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "e5ab782aa1a6446d9b3bfed988be7906"
format_stamp: "Formatted at 2026-08-09 20:29:06 on dist-test-slave-rnrw"
server_key: "9c499f364d78a0ee397fe675031541e9"
server_key_iv: "69e94f4b60aec929389e111309564061"
server_key_version: "encryptionkey@0"
I20260809 20:29:06.043437 2001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:06.061568 2001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:06.066145 2001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:06.066483 2001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:06.087791 2001 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42933
I20260809 20:29:06.087812 2070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42933 every 8 connection(s)
I20260809 20:29:06.089464 2001 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:06.093653 2071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:06.096053 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 2001
I20260809 20:29:06.096256 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:06.096791 12600 external_mini_cluster.cc:1428] Setting key b663b51c67528ac41355cc5f293f6bc3
I20260809 20:29:06.106709 2071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906: Bootstrap starting.
I20260809 20:29:06.110898 2071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:06.112391 2071 log.cc:824] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906: Log is configured to *not* fsync() on all Append() calls
Aug 09 20:29:06 dist-test-slave-rnrw krb5kdc[1986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 20:29:06.115453 2071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906: No bootstrap required, opened a new log
I20260809 20:29:06.121948 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:06.098776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48725 (local address 127.0.0.1:42933)
0809 20:29:06.099759 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:06.099772 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:06.099819 (+ 47us) server_negotiation.cc:407] Connection header received
0809 20:29:06.101772 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:06.101799 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:06.102241 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:06.102621 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:06.104654 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:06.106121 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:06.108110 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:06.108920 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:06.109102 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:06.114220 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:06.114228 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:06.114243 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:06.114280 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:06.117608 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:06.118394 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:06.118400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:06.118407 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:06.118577 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:06.119014 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:06.119026 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:06.119028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:06.119588 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:06.119931 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:06.120372 (+ 441us) server_negotiation.cc:299] Negotiation successful
0809 20:29:06.120870 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":474,"threads_started":1}
I20260809 20:29:06.123826 2071 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab782aa1a6446d9b3bfed988be7906" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } }
I20260809 20:29:06.124255 2071 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:06.124423 2071 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ab782aa1a6446d9b3bfed988be7906, State: Initialized, Role: FOLLOWER
I20260809 20:29:06.125204 2071 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab782aa1a6446d9b3bfed988be7906" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } }
I20260809 20:29:06.125509 2071 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:06.125672 2071 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:06.125885 2071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:06.126034 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:06.128226 2071 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab782aa1a6446d9b3bfed988be7906" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } }
I20260809 20:29:06.128899 2071 leader_election.cc:302] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5ab782aa1a6446d9b3bfed988be7906; no voters:
I20260809 20:29:06.129645 2071 leader_election.cc:288] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:06.129763 2081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:06.132278 2081 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [term 1 LEADER]: Becoming Leader. State: Replica: e5ab782aa1a6446d9b3bfed988be7906, State: Running, Role: LEADER
I20260809 20:29:06.132396 2071 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:06.132961 2081 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab782aa1a6446d9b3bfed988be7906" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } }
I20260809 20:29:06.138303 2082 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5ab782aa1a6446d9b3bfed988be7906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab782aa1a6446d9b3bfed988be7906" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } } }
I20260809 20:29:06.138777 2082 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:06.141261 2083 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5ab782aa1a6446d9b3bfed988be7906. Latest consensus state: current_term: 1 leader_uuid: "e5ab782aa1a6446d9b3bfed988be7906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab782aa1a6446d9b3bfed988be7906" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } } }
I20260809 20:29:06.141716 2083 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:06.144423 2091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:06.149427 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:06.131277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54891 (local address 127.0.0.1:42933)
0809 20:29:06.131656 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:06.131663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:06.131680 (+ 17us) server_negotiation.cc:407] Connection header received
0809 20:29:06.131731 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:06.131737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:06.131805 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:06.131917 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:06.133592 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:06.135136 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:06.136394 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:06.137145 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:06.137301 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:06.140436 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:06.140442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:06.140445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:06.140477 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:06.144811 (+ 4334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:06.145782 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:06.145791 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:06.145798 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:06.146047 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:06.146672 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:06.146676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:06.146680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:06.147361 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:06.147771 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:06.148485 (+ 714us) server_negotiation.cc:299] Negotiation successful
0809 20:29:06.148829 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260809 20:29:06.158507 2091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:06.168466 2091 catalog_manager.cc:1347] Generated new cluster ID: dda4e898ba2541dfb4583e90d817f157
I20260809 20:29:06.168648 2091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:06.217090 2091 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:06.219033 2091 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:06.228760 2091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5ab782aa1a6446d9b3bfed988be7906: Generated new TSK 0
I20260809 20:29:06.229862 2091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:06.244848 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2001
I20260809 20:29:06.246181 2066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:29:06.344607 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 2001
2026-08-09T20:29:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2196 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:06 dist-test-slave-rnrw krb5kdc[2104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:06 dist-test-slave-rnrw krb5kdc[2104](info): set up 2 sockets
Aug 09 20:29:06 dist-test-slave-rnrw krb5kdc[2104](info): commencing operation
krb5kdc: starting...
W20260809 20:29:07.496698 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.006s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:07 dist-test-slave-rnrw krb5kdc[2104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 20:29:07.813541 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:08.163375 2117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:08.163702 2117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:08.163882 2117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:08.171483 2117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:08.171631 2117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:08.171761 2117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:08.171869 2117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:08.171957 2117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:08.182569 2117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:08.183983 2117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:08.185395 2117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:08.198006 2122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:08.198030 2123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:08.198753 2127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:08.199599 2117 server_base.cc:1034] running on GCE node
I20260809 20:29:08.200423 2117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:08.202029 2117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:08.203296 2117 hybrid_clock.cc:648] HybridClock initialized: now 1786307348203255 us; error 47 us; skew 500 ppm
I20260809 20:29:08.203693 2117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:08 dist-test-slave-rnrw krb5kdc[2104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:08.206909 2117 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 20:29:08.208415 2117 webserver.cc:460] Webserver started at http://127.0.0.1:40013/ using document root <none> and password file <none>
I20260809 20:29:08.209177 2117 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:08.209327 2117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:08.209656 2117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:08.212761 2117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "44c43710a0834dcf92f222659380ef36"
format_stamp: "Formatted at 2026-08-09 20:29:08 on dist-test-slave-rnrw"
server_key: "f41d1778c56e08b159cf02e64529bd96"
server_key_iv: "0a0dd6787567cc300726df8d4b39f7f4"
server_key_version: "encryptionkey@0"
I20260809 20:29:08.213587 2117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "44c43710a0834dcf92f222659380ef36"
format_stamp: "Formatted at 2026-08-09 20:29:08 on dist-test-slave-rnrw"
server_key: "f41d1778c56e08b159cf02e64529bd96"
server_key_iv: "0a0dd6787567cc300726df8d4b39f7f4"
server_key_version: "encryptionkey@0"
I20260809 20:29:08.218917 2117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:29:08.222445 2134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:08.223866 2117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 20:29:08.224054 2117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "44c43710a0834dcf92f222659380ef36"
format_stamp: "Formatted at 2026-08-09 20:29:08 on dist-test-slave-rnrw"
server_key: "f41d1778c56e08b159cf02e64529bd96"
server_key_iv: "0a0dd6787567cc300726df8d4b39f7f4"
server_key_version: "encryptionkey@0"
I20260809 20:29:08.224236 2117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:08.242633 2117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:08.247083 2117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:08.247398 2117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:08.266625 2117 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35251
I20260809 20:29:08.266657 2188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35251 every 8 connection(s)
I20260809 20:29:08.268086 2117 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:08.271957 2189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:08.274544 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 2117
I20260809 20:29:08.274749 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:08.275223 12600 external_mini_cluster.cc:1428] Setting key de373d52ef44229b73e528cc6f0397bc
I20260809 20:29:08.284066 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36: Bootstrap starting.
I20260809 20:29:08.287665 2189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:08.288944 2189 log.cc:824] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:08.288950 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:08.277114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42853 (local address 127.0.0.1:35251)
0809 20:29:08.277979 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:08.277990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:08.278031 (+ 41us) server_negotiation.cc:407] Connection header received
0809 20:29:08.279750 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:08.279774 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:08.280171 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:08.280551 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:08.281171 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:08.282515 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:08.283908 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:08.284711 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:08.284897 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:08.286343 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:08.286357 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:08.286372 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:08.286920 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:08.287006 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:08.287038 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:08.287201 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:08.287526 (+ 325us) server_negotiation.cc:299] Negotiation successful
0809 20:29:08.287819 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":391,"threads_started":1}
I20260809 20:29:08.292136 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36: No bootstrap required, opened a new log
I20260809 20:29:08.293084 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:08.301036 2189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c43710a0834dcf92f222659380ef36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35251 } }
I20260809 20:29:08.301621 2189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:08.301751 2189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c43710a0834dcf92f222659380ef36, State: Initialized, Role: FOLLOWER
I20260809 20:29:08.302465 2189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c43710a0834dcf92f222659380ef36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35251 } }
I20260809 20:29:08.302795 2189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:08.302978 2189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:08.303162 2189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:08.305480 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:08.297953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49765 (local address 127.0.0.1:35251)
0809 20:29:08.298173 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:08.298193 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:08.298791 (+ 598us) server_negotiation.cc:407] Connection header received
0809 20:29:08.299010 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:08.299014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:08.299105 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:08.299190 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:08.299942 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:08.300755 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:08.302638 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:08.303325 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:08.303452 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:08.304703 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:08.304708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:08.304711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:08.304731 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:08.304770 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:08.304787 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:08.304868 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:08.305037 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 20:29:08.305139 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 20:29:08.306540 2189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c43710a0834dcf92f222659380ef36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35251 } }
I20260809 20:29:08.307144 2189 leader_election.cc:302] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44c43710a0834dcf92f222659380ef36; no voters:
I20260809 20:29:08.307760 2189 leader_election.cc:288] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:08.307905 2201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:08.308681 2201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [term 1 LEADER]: Becoming Leader. State: Replica: 44c43710a0834dcf92f222659380ef36, State: Running, Role: LEADER
I20260809 20:29:08.309253 2201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c43710a0834dcf92f222659380ef36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35251 } }
I20260809 20:29:08.310061 2189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:08.313499 2204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44c43710a0834dcf92f222659380ef36. Latest consensus state: current_term: 1 leader_uuid: "44c43710a0834dcf92f222659380ef36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c43710a0834dcf92f222659380ef36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35251 } } }
I20260809 20:29:08.313979 2204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:08.314383 2203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44c43710a0834dcf92f222659380ef36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c43710a0834dcf92f222659380ef36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35251 } } }
I20260809 20:29:08.314945 2203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:08.315871 2211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:08.321411 2211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:08.330510 2211 catalog_manager.cc:1347] Generated new cluster ID: 712f6bac193f4463b4d27bd0c9c54a04
I20260809 20:29:08.330648 2211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:08.348253 2211 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:08.349687 2211 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:08.357080 2211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44c43710a0834dcf92f222659380ef36: Generated new TSK 0
I20260809 20:29:08.357908 2211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:08.365233 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2117
W20260809 20:29:08.366318 2184 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42853
W20260809 20:29:08.367156 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2117: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:29:08.367319 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 2117
2026-08-09T20:29:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:08 dist-test-slave-rnrw krb5kdc[2223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:08 dist-test-slave-rnrw krb5kdc[2223](info): set up 2 sockets
Aug 09 20:29:08 dist-test-slave-rnrw krb5kdc[2223](info): commencing operation
krb5kdc: starting...
W20260809 20:29:09.533118 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.004s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:09 dist-test-slave-rnrw krb5kdc[2223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 20:29:09.841764 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:10.190514 2236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:10.190822 2236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:10.190986 2236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:10.198454 2236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:10.198593 2236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:10.198711 2236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:10.198848 2236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:10.198928 2236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:10.209198 2236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:10.210567 2236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:10.211896 2236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:10.224023 2241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:10.223994 2242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:10.224153 2245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:10.224856 2236 server_base.cc:1034] running on GCE node
I20260809 20:29:10.225831 2236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:10.227381 2236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:10.228675 2236 hybrid_clock.cc:648] HybridClock initialized: now 1786307350228631 us; error 53 us; skew 500 ppm
I20260809 20:29:10.229063 2236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:10 dist-test-slave-rnrw krb5kdc[2223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:10.232709 2236 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 20:29:10.234346 2236 webserver.cc:460] Webserver started at http://127.0.0.1:33723/ using document root <none> and password file <none>
I20260809 20:29:10.235088 2236 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:10.235244 2236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:10.235572 2236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:10.238785 2236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "1c62276b1f57421d8838901de6d65fff"
format_stamp: "Formatted at 2026-08-09 20:29:10 on dist-test-slave-rnrw"
server_key: "d2ddca12a11a888ea8cff005917473f4"
server_key_iv: "e20c58047f745ed0bded4d32bb228bf2"
server_key_version: "encryptionkey@0"
I20260809 20:29:10.239629 2236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "1c62276b1f57421d8838901de6d65fff"
format_stamp: "Formatted at 2026-08-09 20:29:10 on dist-test-slave-rnrw"
server_key: "d2ddca12a11a888ea8cff005917473f4"
server_key_iv: "e20c58047f745ed0bded4d32bb228bf2"
server_key_version: "encryptionkey@0"
I20260809 20:29:10.245386 2236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 20:29:10.249296 2252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:10.250852 2236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260809 20:29:10.251085 2236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "1c62276b1f57421d8838901de6d65fff"
format_stamp: "Formatted at 2026-08-09 20:29:10 on dist-test-slave-rnrw"
server_key: "d2ddca12a11a888ea8cff005917473f4"
server_key_iv: "e20c58047f745ed0bded4d32bb228bf2"
server_key_version: "encryptionkey@0"
I20260809 20:29:10.251331 2236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:10.257853 2236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:10.262643 2236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:10.263000 2236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:10.283751 2236 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44607
I20260809 20:29:10.283794 2311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44607 every 8 connection(s)
I20260809 20:29:10.285320 2236 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:10.289371 2312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:10.292234 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 2236
I20260809 20:29:10.292414 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:10.293002 12600 external_mini_cluster.cc:1428] Setting key f8f7e0388b30a2a482e5da2fbb5e59de
I20260809 20:29:10.301193 2315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:10.294892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46703 (local address 127.0.0.1:44607)
0809 20:29:10.295784 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:10.295796 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:10.295840 (+ 44us) server_negotiation.cc:407] Connection header received
0809 20:29:10.297636 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:10.297663 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:10.298044 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:10.298399 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:10.299511 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:10.299517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:10.299533 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:10.299547 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:10.299607 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:10.299630 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:10.299815 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:10.299916 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 20:29:10.300149 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":425,"threads_started":1}
I20260809 20:29:10.302588 2312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff: Bootstrap starting.
I20260809 20:29:10.305186 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:10.306123 2312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:10.307158 2312 log.cc:824] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:10.310302 2312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff: No bootstrap required, opened a new log
I20260809 20:29:10.312479 2315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:10.309839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53037 (local address 127.0.0.1:44607)
0809 20:29:10.310033 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:10.310039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:10.310433 (+ 394us) server_negotiation.cc:407] Connection header received
0809 20:29:10.310647 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:10.310652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:10.310704 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:10.310821 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:10.311776 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:10.311782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:10.311786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:10.311817 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:10.311857 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:10.311874 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:10.312015 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:10.312133 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 20:29:10.312199 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 20:29:10.318614 2312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c62276b1f57421d8838901de6d65fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44607 } }
I20260809 20:29:10.319037 2312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:10.319150 2312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c62276b1f57421d8838901de6d65fff, State: Initialized, Role: FOLLOWER
I20260809 20:29:10.319725 2312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c62276b1f57421d8838901de6d65fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44607 } }
I20260809 20:29:10.319979 2312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:10.320130 2312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:10.320330 2312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:10.323028 2312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c62276b1f57421d8838901de6d65fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44607 } }
I20260809 20:29:10.323555 2312 leader_election.cc:302] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c62276b1f57421d8838901de6d65fff; no voters:
I20260809 20:29:10.324128 2312 leader_election.cc:288] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:10.324288 2322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:10.324999 2322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [term 1 LEADER]: Becoming Leader. State: Replica: 1c62276b1f57421d8838901de6d65fff, State: Running, Role: LEADER
I20260809 20:29:10.325537 2322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c62276b1f57421d8838901de6d65fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44607 } }
I20260809 20:29:10.326356 2312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:10.328601 2323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c62276b1f57421d8838901de6d65fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c62276b1f57421d8838901de6d65fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44607 } } }
I20260809 20:29:10.328722 2324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c62276b1f57421d8838901de6d65fff. Latest consensus state: current_term: 1 leader_uuid: "1c62276b1f57421d8838901de6d65fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c62276b1f57421d8838901de6d65fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44607 } } }
I20260809 20:29:10.329077 2324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:10.329082 2323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:10.332343 2331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:10.338138 2331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:10.347383 2331 catalog_manager.cc:1347] Generated new cluster ID: 2cb9aa65d9a14460846aa50f8a03dbd5
I20260809 20:29:10.347527 2331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:10.359381 2331 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:10.359836 2331 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:10.374142 2331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c62276b1f57421d8838901de6d65fff: Generated new TSK 0
I20260809 20:29:10.374997 2331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:10.395965 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2236
W20260809 20:29:10.397188 2306 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46703
W20260809 20:29:10.397944 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2236: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:29:10.398087 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 2236
2026-08-09T20:29:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2029 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:10 dist-test-slave-rnrw krb5kdc[2345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:10 dist-test-slave-rnrw krb5kdc[2345](info): set up 2 sockets
Aug 09 20:29:10 dist-test-slave-rnrw krb5kdc[2345](info): commencing operation
krb5kdc: starting...
W20260809 20:29:11.560622 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:11 dist-test-slave-rnrw krb5kdc[2345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 20:29:11.864421 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:12.214982 2358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:12.215360 2358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:12.215534 2358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:12.223090 2358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:12.223265 2358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:12.223372 2358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:12.223501 2358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:12.223580 2358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:12.234031 2358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:12.235471 2358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:12.236902 2358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:12.249640 2363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:12.249683 2364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:12.249707 2366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:12.250831 2358 server_base.cc:1034] running on GCE node
I20260809 20:29:12.251654 2358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:12.253216 2358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:12.254441 2358 hybrid_clock.cc:648] HybridClock initialized: now 1786307352254373 us; error 54 us; skew 500 ppm
I20260809 20:29:12.254786 2358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:12 dist-test-slave-rnrw krb5kdc[2345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:12.264122 2358 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 20:29:12.266166 2358 webserver.cc:460] Webserver started at http://127.0.0.1:42739/ using document root <none> and password file <none>
I20260809 20:29:12.267004 2358 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:12.267161 2358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:12.267567 2358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:12.271134 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "b5b78c5bd5d643e1ba3f28456bd8e633"
format_stamp: "Formatted at 2026-08-09 20:29:12 on dist-test-slave-rnrw"
server_key: "b94c47b2d54c43807e4812f6f54bccd0"
server_key_iv: "ee810bb07bd19fb1fa90a85b51dd17d9"
server_key_version: "encryptionkey@0"
I20260809 20:29:12.272055 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "b5b78c5bd5d643e1ba3f28456bd8e633"
format_stamp: "Formatted at 2026-08-09 20:29:12 on dist-test-slave-rnrw"
server_key: "b94c47b2d54c43807e4812f6f54bccd0"
server_key_iv: "ee810bb07bd19fb1fa90a85b51dd17d9"
server_key_version: "encryptionkey@0"
I20260809 20:29:12.278214 2358 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260809 20:29:12.282353 2373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:12.283898 2358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 20:29:12.284121 2358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "b5b78c5bd5d643e1ba3f28456bd8e633"
format_stamp: "Formatted at 2026-08-09 20:29:12 on dist-test-slave-rnrw"
server_key: "b94c47b2d54c43807e4812f6f54bccd0"
server_key_iv: "ee810bb07bd19fb1fa90a85b51dd17d9"
server_key_version: "encryptionkey@0"
I20260809 20:29:12.284348 2358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:12.299095 2358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:12.303422 2358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:12.303750 2358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:12.323475 2358 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41813
I20260809 20:29:12.323515 2429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41813 every 8 connection(s)
I20260809 20:29:12.325004 2358 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:12.328794 2430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:12.335198 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 2358
I20260809 20:29:12.335417 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:12.336009 12600 external_mini_cluster.cc:1428] Setting key 93666d98ff6669aa546238dcdf61e6fa
I20260809 20:29:12.340198 2430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633: Bootstrap starting.
I20260809 20:29:12.344188 2430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:12.345515 2430 log.cc:824] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:12.349028 2430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633: No bootstrap required, opened a new log
Aug 09 20:29:12 dist-test-slave-rnrw krb5kdc[2345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 20:29:12.357892 2430 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b78c5bd5d643e1ba3f28456bd8e633" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } }
I20260809 20:29:12.358322 2430 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:12.358450 2430 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b78c5bd5d643e1ba3f28456bd8e633, State: Initialized, Role: FOLLOWER
I20260809 20:29:12.359120 2430 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b78c5bd5d643e1ba3f28456bd8e633" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } }
I20260809 20:29:12.359391 2430 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:12.359565 2430 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:12.359849 2430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:12.361088 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:12.337979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:55255 (local address 127.0.0.1:41813)
0809 20:29:12.338816 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:12.338831 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:12.338874 (+ 43us) server_negotiation.cc:407] Connection header received
0809 20:29:12.340777 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:12.340803 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:12.341203 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:12.341524 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:12.343490 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:12.344998 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:12.346774 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:12.347515 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:12.347650 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:12.352946 (+ 5296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:12.352960 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:12.352986 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:12.353031 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:12.356384 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:12.357307 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:12.357314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:12.357321 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:12.357497 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:12.357972 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:12.357976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:12.357978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:12.358665 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:12.359029 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:12.359515 (+ 486us) server_negotiation.cc:299] Negotiation successful
0809 20:29:12.359867 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":371,"threads_started":1}
I20260809 20:29:12.362514 2430 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b78c5bd5d643e1ba3f28456bd8e633" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } }
I20260809 20:29:12.363101 2430 leader_election.cc:302] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5b78c5bd5d643e1ba3f28456bd8e633; no voters:
I20260809 20:29:12.363829 2430 leader_election.cc:288] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:12.363919 2436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:12.365150 2436 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [term 1 LEADER]: Becoming Leader. State: Replica: b5b78c5bd5d643e1ba3f28456bd8e633, State: Running, Role: LEADER
I20260809 20:29:12.365373 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:12.365797 2436 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b78c5bd5d643e1ba3f28456bd8e633" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } }
I20260809 20:29:12.366153 2430 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:12.370571 2437 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5b78c5bd5d643e1ba3f28456bd8e633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b78c5bd5d643e1ba3f28456bd8e633" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } } }
I20260809 20:29:12.371165 2437 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:12.371106 2438 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5b78c5bd5d643e1ba3f28456bd8e633. Latest consensus state: current_term: 1 leader_uuid: "b5b78c5bd5d643e1ba3f28456bd8e633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b78c5bd5d643e1ba3f28456bd8e633" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } } }
I20260809 20:29:12.371449 2438 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:12.372402 2446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:12.380788 2446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:12.405916 2446 catalog_manager.cc:1347] Generated new cluster ID: 7116a13b2beb44a1ad388379d61ef81c
I20260809 20:29:12.406044 2446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:12.407456 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:12.388115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:52739 (local address 127.0.0.1:41813)
0809 20:29:12.388393 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:12.388400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:12.388525 (+ 125us) server_negotiation.cc:407] Connection header received
0809 20:29:12.388815 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:12.388820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:12.388906 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:12.389024 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:12.391077 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:12.392258 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:12.394208 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:12.394916 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:12.395016 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:12.398708 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:12.398716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:12.398722 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:12.398764 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:12.403206 (+ 4442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:12.404729 (+ 1523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:12.404735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:12.404738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:12.404894 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:12.405518 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:12.405523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:12.405526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:12.405974 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:12.406251 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:12.406833 (+ 582us) server_negotiation.cc:299] Negotiation successful
0809 20:29:12.407042 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 20:29:12.417873 2446 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:12.419301 2446 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:12.426040 2446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5b78c5bd5d643e1ba3f28456bd8e633: Generated new TSK 0
I20260809 20:29:12.426901 2446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:12.432909 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2358
I20260809 20:29:12.434242 2424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:29:12.525544 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 2358
W20260809 20:29:12.545280 2341 connection.cc:535] client connection to 127.0.0.1:41813 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41813): Connection reset by peer (error 104)
2026-08-09T20:29:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2126 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:12 dist-test-slave-rnrw krb5kdc[2464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:12 dist-test-slave-rnrw krb5kdc[2464](info): set up 2 sockets
Aug 09 20:29:12 dist-test-slave-rnrw krb5kdc[2464](info): commencing operation
krb5kdc: starting...
W20260809 20:29:13.680225 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:13 dist-test-slave-rnrw krb5kdc[2464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 20:29:13.994352 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:14.341890 2478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:14.342249 2478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:14.342410 2478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:14.349998 2478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:14.350179 2478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:14.350291 2478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:14.350402 2478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:14.350482 2478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:14.360637 2478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:14.362010 2478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:14.363322 2478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:14.375811 2486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:14.375813 2484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:14.376296 2483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:14.376833 2478 server_base.cc:1034] running on GCE node
I20260809 20:29:14.377722 2478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:14.379164 2478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:14.380414 2478 hybrid_clock.cc:648] HybridClock initialized: now 1786307354380347 us; error 55 us; skew 500 ppm
I20260809 20:29:14.380820 2478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:14 dist-test-slave-rnrw krb5kdc[2464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:14.384207 2478 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 20:29:14.385810 2478 webserver.cc:460] Webserver started at http://127.0.0.1:40115/ using document root <none> and password file <none>
I20260809 20:29:14.386518 2478 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:14.386679 2478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:14.386996 2478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:14.390138 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "ac3da0c4218b433cb9530d1a73b707ef"
format_stamp: "Formatted at 2026-08-09 20:29:14 on dist-test-slave-rnrw"
server_key: "3fb1fec97e46cfb05b4ca3a926c6b1a2"
server_key_iv: "2818b7d2513184d6e6981a706d3ce963"
server_key_version: "encryptionkey@0"
I20260809 20:29:14.390944 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "ac3da0c4218b433cb9530d1a73b707ef"
format_stamp: "Formatted at 2026-08-09 20:29:14 on dist-test-slave-rnrw"
server_key: "3fb1fec97e46cfb05b4ca3a926c6b1a2"
server_key_iv: "2818b7d2513184d6e6981a706d3ce963"
server_key_version: "encryptionkey@0"
I20260809 20:29:14.396384 2478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 20:29:14.400044 2493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:14.401425 2478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:29:14.401625 2478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "ac3da0c4218b433cb9530d1a73b707ef"
format_stamp: "Formatted at 2026-08-09 20:29:14 on dist-test-slave-rnrw"
server_key: "3fb1fec97e46cfb05b4ca3a926c6b1a2"
server_key_iv: "2818b7d2513184d6e6981a706d3ce963"
server_key_version: "encryptionkey@0"
I20260809 20:29:14.401829 2478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:14.442480 2478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:14.447288 2478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:14.447639 2478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:14.467114 2478 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41251
I20260809 20:29:14.467150 2548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41251 every 8 connection(s)
I20260809 20:29:14.468636 2478 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:14.472365 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:14.476456 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 2478
I20260809 20:29:14.476689 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:14.477294 12600 external_mini_cluster.cc:1428] Setting key 159bd4e3546ce59a716689830cec9b88
I20260809 20:29:14.485986 2549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef: Bootstrap starting.
I20260809 20:29:14.489835 2549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:14.491047 2549 log.cc:824] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:14.491263 2553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:14.479270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:40127 (local address 127.0.0.1:41251)
0809 20:29:14.480071 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:14.480083 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:14.480130 (+ 47us) server_negotiation.cc:407] Connection header received
0809 20:29:14.481865 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:14.481890 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:14.482346 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:14.482690 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:14.483338 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:14.484735 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:14.486723 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:14.487390 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:14.487543 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:14.488928 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:14.488936 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:14.488949 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:14.489498 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:14.489576 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:14.489605 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:14.489733 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:14.490123 (+ 390us) server_negotiation.cc:299] Negotiation successful
0809 20:29:14.490263 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":388,"threads_started":1}
I20260809 20:29:14.494693 2549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef: No bootstrap required, opened a new log
I20260809 20:29:14.495638 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:14.503855 2549 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3da0c4218b433cb9530d1a73b707ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } }
I20260809 20:29:14.504323 2549 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:14.504462 2549 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3da0c4218b433cb9530d1a73b707ef, State: Initialized, Role: FOLLOWER
I20260809 20:29:14.505221 2549 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3da0c4218b433cb9530d1a73b707ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } }
I20260809 20:29:14.505561 2549 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:14.505719 2549 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:14.505923 2549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:14.507355 2553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:14.500185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:44141 (local address 127.0.0.1:41251)
0809 20:29:14.500382 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:14.500390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:14.500975 (+ 585us) server_negotiation.cc:407] Connection header received
0809 20:29:14.501154 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:14.501160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:14.501250 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:14.501382 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:14.501889 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:14.503086 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:14.504489 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:14.505055 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:14.505204 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:14.506515 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:14.506529 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:14.506533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:14.506552 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:14.506615 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:14.506642 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:14.506781 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:14.506993 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 20:29:14.507053 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 20:29:14.509298 2549 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3da0c4218b433cb9530d1a73b707ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } }
I20260809 20:29:14.509837 2549 leader_election.cc:302] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3da0c4218b433cb9530d1a73b707ef; no voters:
I20260809 20:29:14.510456 2549 leader_election.cc:288] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:14.510582 2561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:14.511464 2561 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [term 1 LEADER]: Becoming Leader. State: Replica: ac3da0c4218b433cb9530d1a73b707ef, State: Running, Role: LEADER
I20260809 20:29:14.512101 2561 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3da0c4218b433cb9530d1a73b707ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } }
I20260809 20:29:14.512873 2549 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:14.514966 2562 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac3da0c4218b433cb9530d1a73b707ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3da0c4218b433cb9530d1a73b707ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } } }
I20260809 20:29:14.514966 2563 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac3da0c4218b433cb9530d1a73b707ef. Latest consensus state: current_term: 1 leader_uuid: "ac3da0c4218b433cb9530d1a73b707ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3da0c4218b433cb9530d1a73b707ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } } }
I20260809 20:29:14.515460 2562 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:14.515460 2563 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:14.521762 2567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:14.527278 2567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:14.536804 2567 catalog_manager.cc:1347] Generated new cluster ID: aaea0ee2258047fd94adeeca70aef340
I20260809 20:29:14.536942 2567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:14.555949 2567 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:14.557302 2567 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:14.568472 2567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac3da0c4218b433cb9530d1a73b707ef: Generated new TSK 0
I20260809 20:29:14.569432 2567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:14.580015 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2478
W20260809 20:29:14.581288 2544 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.60:40127
W20260809 20:29:14.582203 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2478: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:29:14.582368 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 2478
W20260809 20:29:14.602113 2462 connection.cc:535] client connection to 127.0.0.1:41251 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41251): Connection reset by peer (error 104)
2026-08-09T20:29:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2056 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:14 dist-test-slave-rnrw krb5kdc[2585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:14 dist-test-slave-rnrw krb5kdc[2585](info): set up 2 sockets
Aug 09 20:29:14 dist-test-slave-rnrw krb5kdc[2585](info): commencing operation
krb5kdc: starting...
W20260809 20:29:15.748924 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:15 dist-test-slave-rnrw krb5kdc[2585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 20:29:16.059653 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:16.403507 2599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:16.403864 2599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:16.404029 2599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:16.411429 2599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:16.411607 2599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:16.411715 2599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:16.411821 2599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:16.411899 2599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:16.423205 2599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:16.424643 2599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:16.425992 2599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:16.438478 2604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:16.438527 2607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:16.438469 2605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:16.439357 2599 server_base.cc:1034] running on GCE node
I20260809 20:29:16.440256 2599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:16.441821 2599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:16.443121 2599 hybrid_clock.cc:648] HybridClock initialized: now 1786307356443050 us; error 59 us; skew 500 ppm
I20260809 20:29:16.443606 2599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:16 dist-test-slave-rnrw krb5kdc[2585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:16.447006 2599 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 20:29:16.448657 2599 webserver.cc:460] Webserver started at http://127.0.0.1:43303/ using document root <none> and password file <none>
I20260809 20:29:16.449378 2599 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:16.449536 2599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:16.449883 2599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:16.453408 2599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "5df3d283971245909f60ef887255beb5"
format_stamp: "Formatted at 2026-08-09 20:29:16 on dist-test-slave-rnrw"
server_key: "d667fd900983aa960a1a4153551e13f8"
server_key_iv: "b4d2487e96db3ce957355891de86559e"
server_key_version: "encryptionkey@0"
I20260809 20:29:16.454228 2599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "5df3d283971245909f60ef887255beb5"
format_stamp: "Formatted at 2026-08-09 20:29:16 on dist-test-slave-rnrw"
server_key: "d667fd900983aa960a1a4153551e13f8"
server_key_iv: "b4d2487e96db3ce957355891de86559e"
server_key_version: "encryptionkey@0"
I20260809 20:29:16.459470 2599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:29:16.463119 2614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:16.464447 2599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 20:29:16.464702 2599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "5df3d283971245909f60ef887255beb5"
format_stamp: "Formatted at 2026-08-09 20:29:16 on dist-test-slave-rnrw"
server_key: "d667fd900983aa960a1a4153551e13f8"
server_key_iv: "b4d2487e96db3ce957355891de86559e"
server_key_version: "encryptionkey@0"
I20260809 20:29:16.464911 2599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:16.470753 2599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:16.474861 2599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:16.475179 2599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:16.494436 2599 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33669
I20260809 20:29:16.494479 2668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33669 every 8 connection(s)
I20260809 20:29:16.495975 2599 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:16.498353 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 2599
I20260809 20:29:16.498558 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:16.499135 12600 external_mini_cluster.cc:1428] Setting key fc4dd7ba23a980bc20306b797f3439d2
I20260809 20:29:16.500789 2669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:16.507094 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:16.501061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:33515 (local address 127.0.0.1:33669)
0809 20:29:16.501895 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:16.501907 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:16.501949 (+ 42us) server_negotiation.cc:407] Connection header received
0809 20:29:16.503564 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:16.503588 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:16.503934 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:16.504352 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:16.505454 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:16.505459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:16.505475 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:16.505492 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:16.505561 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:16.505582 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:16.505821 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:16.505965 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 20:29:16.506164 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":334,"threads_started":1}
I20260809 20:29:16.510886 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:16.513407 2669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5: Bootstrap starting.
I20260809 20:29:16.517606 2669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:16.517990 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:16.515516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:56961 (local address 127.0.0.1:33669)
0809 20:29:16.515718 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:16.515725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:16.515999 (+ 274us) server_negotiation.cc:407] Connection header received
0809 20:29:16.516167 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:16.516173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:16.516240 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:16.516348 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:16.517313 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:16.517321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:16.517327 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:16.517349 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:16.517404 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:16.517430 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:16.517566 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:16.517635 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 20:29:16.517699 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 20:29:16.518929 2669 log.cc:824] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:16.522110 2669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5: No bootstrap required, opened a new log
I20260809 20:29:16.530591 2669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df3d283971245909f60ef887255beb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } }
I20260809 20:29:16.531025 2669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:16.531153 2669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df3d283971245909f60ef887255beb5, State: Initialized, Role: FOLLOWER
I20260809 20:29:16.531767 2669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df3d283971245909f60ef887255beb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } }
I20260809 20:29:16.531993 2669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:16.532158 2669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:16.532335 2669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:16.534994 2669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df3d283971245909f60ef887255beb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } }
I20260809 20:29:16.535516 2669 leader_election.cc:302] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5df3d283971245909f60ef887255beb5; no voters:
I20260809 20:29:16.536101 2669 leader_election.cc:288] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:16.536250 2680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:16.536978 2680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [term 1 LEADER]: Becoming Leader. State: Replica: 5df3d283971245909f60ef887255beb5, State: Running, Role: LEADER
I20260809 20:29:16.537539 2680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df3d283971245909f60ef887255beb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } }
I20260809 20:29:16.538292 2669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:16.540419 2682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5df3d283971245909f60ef887255beb5. Latest consensus state: current_term: 1 leader_uuid: "5df3d283971245909f60ef887255beb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df3d283971245909f60ef887255beb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } } }
I20260809 20:29:16.540906 2682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:16.540758 2681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5df3d283971245909f60ef887255beb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df3d283971245909f60ef887255beb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } } }
I20260809 20:29:16.541047 2681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:16.542485 2687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:16.548218 2687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:16.558588 2687 catalog_manager.cc:1347] Generated new cluster ID: 2ad3b7808e07401c897bbf7d83264005
I20260809 20:29:16.558707 2687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:16.585420 2687 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:16.585881 2687 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:16.597594 2687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5df3d283971245909f60ef887255beb5: Generated new TSK 0
I20260809 20:29:16.598666 2687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:16.618146 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2599
W20260809 20:29:16.619755 2664 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.60:33515
W20260809 20:29:16.620565 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2599: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:29:16.620720 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 2599
2026-08-09T20:29:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2038 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:16 dist-test-slave-rnrw krb5kdc[2702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:16 dist-test-slave-rnrw krb5kdc[2702](info): set up 2 sockets
Aug 09 20:29:16 dist-test-slave-rnrw krb5kdc[2702](info): commencing operation
krb5kdc: starting...
W20260809 20:29:17.780205 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:17 dist-test-slave-rnrw krb5kdc[2702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 20:29:18.093923 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40935
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:18.443789 2718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:18.444095 2718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:18.444192 2718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:18.451589 2718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:18.451709 2718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:18.451758 2718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:18.451828 2718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:18.451872 2718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:18.461805 2718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40935
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:18.463055 2718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:18.464310 2718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:18.476595 2726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:18.476672 2723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:18.476624 2724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:18.477460 2718 server_base.cc:1034] running on GCE node
I20260809 20:29:18.478189 2718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:18.479590 2718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:18.480794 2718 hybrid_clock.cc:648] HybridClock initialized: now 1786307358480746 us; error 40 us; skew 500 ppm
I20260809 20:29:18.481146 2718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:18 dist-test-slave-rnrw krb5kdc[2702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:18.487085 2718 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 20:29:18.488795 2718 webserver.cc:460] Webserver started at http://127.0.0.1:45749/ using document root <none> and password file <none>
I20260809 20:29:18.489490 2718 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:18.489646 2718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:18.489961 2718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:18.493219 2718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "4c0a593f36a84cdea34d98837cd92f1a"
format_stamp: "Formatted at 2026-08-09 20:29:18 on dist-test-slave-rnrw"
server_key: "f3531309c06e021d653495544d7a345e"
server_key_iv: "9eaa061d53d492e4dd5b34d539f2cde3"
server_key_version: "encryptionkey@0"
I20260809 20:29:18.494050 2718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "4c0a593f36a84cdea34d98837cd92f1a"
format_stamp: "Formatted at 2026-08-09 20:29:18 on dist-test-slave-rnrw"
server_key: "f3531309c06e021d653495544d7a345e"
server_key_iv: "9eaa061d53d492e4dd5b34d539f2cde3"
server_key_version: "encryptionkey@0"
I20260809 20:29:18.499431 2718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 20:29:18.503134 2734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:18.504653 2718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 20:29:18.504844 2718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "4c0a593f36a84cdea34d98837cd92f1a"
format_stamp: "Formatted at 2026-08-09 20:29:18 on dist-test-slave-rnrw"
server_key: "f3531309c06e021d653495544d7a345e"
server_key_iv: "9eaa061d53d492e4dd5b34d539f2cde3"
server_key_version: "encryptionkey@0"
I20260809 20:29:18.505045 2718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:18.519996 2718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:18.524345 2718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:18.524729 2718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:18.544157 2718 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40935
I20260809 20:29:18.544317 2787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40935 every 8 connection(s)
I20260809 20:29:18.545605 2718 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:18.549492 2788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:18.554809 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 2718
I20260809 20:29:18.555006 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:18.555464 12600 external_mini_cluster.cc:1428] Setting key d9793923ea4428374f1ebf7e67501e74
I20260809 20:29:18.560123 2788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a: Bootstrap starting.
I20260809 20:29:18.563731 2788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:18.565024 2788 log.cc:824] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:18.568470 2788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a: No bootstrap required, opened a new log
Aug 09 20:29:18 dist-test-slave-rnrw krb5kdc[2702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 20:29:18.579142 2788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0a593f36a84cdea34d98837cd92f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } }
I20260809 20:29:18.579648 2788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:18.579771 2788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0a593f36a84cdea34d98837cd92f1a, State: Initialized, Role: FOLLOWER
I20260809 20:29:18.580469 2788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0a593f36a84cdea34d98837cd92f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } }
I20260809 20:29:18.580794 2788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:18.580962 2788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:18.581001 2792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:18.557249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:55091 (local address 127.0.0.1:40935)
0809 20:29:18.558082 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:18.558093 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:18.558135 (+ 42us) server_negotiation.cc:407] Connection header received
0809 20:29:18.559789 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:18.559812 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:18.560180 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:18.560532 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:18.562370 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:18.563667 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:18.565492 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:18.566616 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:18.566741 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:18.571688 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:18.571700 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:18.571723 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:18.571773 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:18.576035 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:18.577026 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:18.577037 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:18.577051 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:18.577284 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:18.577730 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:18.577735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:18.577738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:18.578435 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:18.578776 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:18.579183 (+ 407us) server_negotiation.cc:299] Negotiation successful
0809 20:29:18.579547 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":391,"threads_started":1}
I20260809 20:29:18.581177 2788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:18.583566 2788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0a593f36a84cdea34d98837cd92f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } }
I20260809 20:29:18.584177 2788 leader_election.cc:302] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c0a593f36a84cdea34d98837cd92f1a; no voters:
I20260809 20:29:18.584879 2788 leader_election.cc:288] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:18.584957 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:18.585027 2794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:18.586896 2788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:18.594179 2794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [term 1 LEADER]: Becoming Leader. State: Replica: 4c0a593f36a84cdea34d98837cd92f1a, State: Running, Role: LEADER
I20260809 20:29:18.594985 2794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0a593f36a84cdea34d98837cd92f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } }
I20260809 20:29:18.598253 2796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c0a593f36a84cdea34d98837cd92f1a. Latest consensus state: current_term: 1 leader_uuid: "4c0a593f36a84cdea34d98837cd92f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0a593f36a84cdea34d98837cd92f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } } }
I20260809 20:29:18.598722 2796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:18.599257 2795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c0a593f36a84cdea34d98837cd92f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0a593f36a84cdea34d98837cd92f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } } }
I20260809 20:29:18.599491 2795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:18.601313 2811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:18.606949 2811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:18.614439 2792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:18.594727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:44973 (local address 127.0.0.1:40935)
0809 20:29:18.594952 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:18.594965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:18.595694 (+ 729us) server_negotiation.cc:407] Connection header received
0809 20:29:18.595935 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:18.595940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:18.596022 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:18.596141 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:18.598462 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:18.599747 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:18.601590 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:18.602345 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:18.602513 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:18.605772 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:18.605780 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:18.605786 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:18.605830 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:18.609925 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:18.611141 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:18.611149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:18.611155 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:18.611313 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:18.611945 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:18.611949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:18.611952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:18.612372 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:18.612658 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:18.613869 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0809 20:29:18.614046 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 20:29:18.623878 2811 catalog_manager.cc:1347] Generated new cluster ID: 8393e47fdc9c444fa05a059226c8ebaf
I20260809 20:29:18.623997 2811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:18.633926 2811 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:18.635807 2811 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:18.646804 2811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c0a593f36a84cdea34d98837cd92f1a: Generated new TSK 0
I20260809 20:29:18.647914 2811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:18.657652 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2718
I20260809 20:29:18.659022 2783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:29:18.748690 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 2718
W20260809 20:29:18.768939 2700 connection.cc:535] client connection to 127.0.0.1:40935 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40935): Connection reset by peer (error 104)
2026-08-09T20:29:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2134 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:18 dist-test-slave-rnrw krb5kdc[2821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:18 dist-test-slave-rnrw krb5kdc[2821](info): set up 2 sockets
Aug 09 20:29:18 dist-test-slave-rnrw krb5kdc[2821](info): commencing operation
krb5kdc: starting...
W20260809 20:29:19.951802 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.001s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:20 dist-test-slave-rnrw krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 20:29:20.253396 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34577
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:20.604931 2835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:20.605228 2835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:20.605422 2835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:20.613111 2835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:20.613229 2835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:20.613343 2835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:20.613478 2835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:20.613539 2835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:20.623700 2835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34577
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:20.625087 2835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:20.626395 2835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:20.638545 2841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:20.638586 2844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:20.638634 2842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:20.639475 2835 server_base.cc:1034] running on GCE node
I20260809 20:29:20.640192 2835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:20.641578 2835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:20.642776 2835 hybrid_clock.cc:648] HybridClock initialized: now 1786307360642721 us; error 42 us; skew 500 ppm
I20260809 20:29:20.643077 2835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:20 dist-test-slave-rnrw krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:20.646301 2835 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 20:29:20.647945 2835 webserver.cc:460] Webserver started at http://127.0.0.1:41047/ using document root <none> and password file <none>
I20260809 20:29:20.648746 2835 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:20.648906 2835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:20.649225 2835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:20.652268 2835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "0645e41aea8f44b789cf5b3196d472a9"
format_stamp: "Formatted at 2026-08-09 20:29:20 on dist-test-slave-rnrw"
server_key: "eb27c15a606dceb29de995054f5e923c"
server_key_iv: "518fc76b237207296ef29c954595ed15"
server_key_version: "encryptionkey@0"
I20260809 20:29:20.653129 2835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "0645e41aea8f44b789cf5b3196d472a9"
format_stamp: "Formatted at 2026-08-09 20:29:20 on dist-test-slave-rnrw"
server_key: "eb27c15a606dceb29de995054f5e923c"
server_key_iv: "518fc76b237207296ef29c954595ed15"
server_key_version: "encryptionkey@0"
I20260809 20:29:20.658368 2835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:29:20.661870 2851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:20.663180 2835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:29:20.663372 2835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "0645e41aea8f44b789cf5b3196d472a9"
format_stamp: "Formatted at 2026-08-09 20:29:20 on dist-test-slave-rnrw"
server_key: "eb27c15a606dceb29de995054f5e923c"
server_key_iv: "518fc76b237207296ef29c954595ed15"
server_key_version: "encryptionkey@0"
I20260809 20:29:20.663574 2835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:20.683878 2835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:20.688385 2835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:20.688735 2835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:20.708297 2835 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34577
I20260809 20:29:20.708307 2904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34577 every 8 connection(s)
I20260809 20:29:20.709739 2835 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:20.713447 2905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:20.713734 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 2835
I20260809 20:29:20.713924 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:20.714498 12600 external_mini_cluster.cc:1428] Setting key c10deb704a47e498b7c3bf2f6574b816
I20260809 20:29:20.727476 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:20.716278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:54011 (local address 127.0.0.1:34577)
0809 20:29:20.717116 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:20.717126 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:20.717168 (+ 42us) server_negotiation.cc:407] Connection header received
0809 20:29:20.718619 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:20.718640 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:20.719044 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:20.719357 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:20.719953 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:20.721372 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:20.723131 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:20.723795 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:20.723921 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:20.725247 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:20.725257 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:20.725268 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:20.725798 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:20.725894 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:20.725917 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:20.726036 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:20.726348 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 20:29:20.726520 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":414,"threads_started":1}
I20260809 20:29:20.727641 2905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9: Bootstrap starting.
I20260809 20:29:20.730870 2905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:20.731225 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:20.731972 2905 log.cc:824] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:20.734905 2905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9: No bootstrap required, opened a new log
I20260809 20:29:20.742985 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:20.736123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:54605 (local address 127.0.0.1:34577)
0809 20:29:20.736327 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:20.736333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:20.736645 (+ 312us) server_negotiation.cc:407] Connection header received
0809 20:29:20.736838 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:20.736842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:20.736910 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:20.736990 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:20.737605 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:20.738394 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:20.740354 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:20.740875 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:20.740970 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:20.742192 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:20.742197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:20.742203 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:20.742223 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:20.742258 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:20.742272 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:20.742353 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:20.742551 (+ 198us) server_negotiation.cc:299] Negotiation successful
0809 20:29:20.742597 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 20:29:20.745206 2905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0645e41aea8f44b789cf5b3196d472a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34577 } }
I20260809 20:29:20.745602 2905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:20.745716 2905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0645e41aea8f44b789cf5b3196d472a9, State: Initialized, Role: FOLLOWER
I20260809 20:29:20.746409 2905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0645e41aea8f44b789cf5b3196d472a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34577 } }
I20260809 20:29:20.746707 2905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:20.746855 2905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:20.746999 2905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:20.749233 2905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0645e41aea8f44b789cf5b3196d472a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34577 } }
I20260809 20:29:20.749666 2905 leader_election.cc:302] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0645e41aea8f44b789cf5b3196d472a9; no voters:
I20260809 20:29:20.750228 2905 leader_election.cc:288] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:20.750376 2915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:20.751140 2915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [term 1 LEADER]: Becoming Leader. State: Replica: 0645e41aea8f44b789cf5b3196d472a9, State: Running, Role: LEADER
I20260809 20:29:20.751726 2915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0645e41aea8f44b789cf5b3196d472a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34577 } }
I20260809 20:29:20.752591 2905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:20.754870 2917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0645e41aea8f44b789cf5b3196d472a9. Latest consensus state: current_term: 1 leader_uuid: "0645e41aea8f44b789cf5b3196d472a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0645e41aea8f44b789cf5b3196d472a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34577 } } }
I20260809 20:29:20.755276 2917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:20.755894 2916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0645e41aea8f44b789cf5b3196d472a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0645e41aea8f44b789cf5b3196d472a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34577 } } }
I20260809 20:29:20.756371 2916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:20.757884 2925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:20.763264 2925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:20.763978 2934 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 20:29:20.772107 2925 catalog_manager.cc:1347] Generated new cluster ID: fa4aa8cebcea463c89f842c0d159d05c
I20260809 20:29:20.772218 2925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:20.792292 2925 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:20.794193 2925 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:20.805023 2925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0645e41aea8f44b789cf5b3196d472a9: Generated new TSK 0
I20260809 20:29:20.806144 2925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:20.811718 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2835
W20260809 20:29:20.813489 2899 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.60:54011
W20260809 20:29:20.814257 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2835: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:29:20.814409 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 2835
W20260809 20:29:20.833784 2819 connection.cc:535] client connection to 127.0.0.1:34577 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34577): Connection reset by peer (error 104)
2026-08-09T20:29:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2059 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:20 dist-test-slave-rnrw krb5kdc[2938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:20 dist-test-slave-rnrw krb5kdc[2938](info): set up 2 sockets
Aug 09 20:29:20 dist-test-slave-rnrw krb5kdc[2938](info): commencing operation
krb5kdc: starting...
W20260809 20:29:21.974131 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:22 dist-test-slave-rnrw krb5kdc[2938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 20:29:22.286067 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33659
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:22.633961 2953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:22.634256 2953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:22.634382 2953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:22.641618 2953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:22.641755 2953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:22.641863 2953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:22.641971 2953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:22.642052 2953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:22.651975 2953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:22.653335 2953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:22.654678 2953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:22.666594 2958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:22.666559 2959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:22.666654 2961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:22.667773 2953 server_base.cc:1034] running on GCE node
I20260809 20:29:22.668639 2953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:22.670066 2953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:22.671274 2953 hybrid_clock.cc:648] HybridClock initialized: now 1786307362671238 us; error 31 us; skew 500 ppm
I20260809 20:29:22.671648 2953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:22 dist-test-slave-rnrw krb5kdc[2938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:22.674914 2953 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 20:29:22.676473 2953 webserver.cc:460] Webserver started at http://127.0.0.1:40561/ using document root <none> and password file <none>
I20260809 20:29:22.677212 2953 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:22.677366 2953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:22.677685 2953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:22.680735 2953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "a263e88cbe3d41cea1bd2e65e04e6e61"
format_stamp: "Formatted at 2026-08-09 20:29:22 on dist-test-slave-rnrw"
server_key: "b1bb48da06dca6abbc9d04ae66d57e8a"
server_key_iv: "b8a70476443d8ff7e46ecb9fb7bd4c20"
server_key_version: "encryptionkey@0"
I20260809 20:29:22.681519 2953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "a263e88cbe3d41cea1bd2e65e04e6e61"
format_stamp: "Formatted at 2026-08-09 20:29:22 on dist-test-slave-rnrw"
server_key: "b1bb48da06dca6abbc9d04ae66d57e8a"
server_key_iv: "b8a70476443d8ff7e46ecb9fb7bd4c20"
server_key_version: "encryptionkey@0"
I20260809 20:29:22.686689 2953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:29:22.690057 2969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:22.691337 2953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 20:29:22.691525 2953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "a263e88cbe3d41cea1bd2e65e04e6e61"
format_stamp: "Formatted at 2026-08-09 20:29:22 on dist-test-slave-rnrw"
server_key: "b1bb48da06dca6abbc9d04ae66d57e8a"
server_key_iv: "b8a70476443d8ff7e46ecb9fb7bd4c20"
server_key_version: "encryptionkey@0"
I20260809 20:29:22.691740 2953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:22.697500 2953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:22.701745 2953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:22.702046 2953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:22.721367 2953 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33659
I20260809 20:29:22.721450 3020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33659 every 8 connection(s)
I20260809 20:29:22.722779 2953 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:22.726403 3022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:22.726692 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 2953
I20260809 20:29:22.726876 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:22.727308 12600 external_mini_cluster.cc:1428] Setting key 9b9162f02cf68c8196b72e844cff54a0
I20260809 20:29:22.735437 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:22.729147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:51131 (local address 127.0.0.1:33659)
0809 20:29:22.730022 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:22.730038 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:22.730084 (+ 46us) server_negotiation.cc:407] Connection header received
0809 20:29:22.731675 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:22.731711 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:22.732132 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:22.732580 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:22.733550 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:22.733560 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:22.733582 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:22.733604 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:22.733679 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:22.733720 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:22.733944 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:22.734079 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 20:29:22.734262 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":347,"threads_started":1}
I20260809 20:29:22.739084 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:22.740785 3022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61: Bootstrap starting.
I20260809 20:29:22.744082 3022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:22.745316 3022 log.cc:824] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:22.746243 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:22.743514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:48781 (local address 127.0.0.1:33659)
0809 20:29:22.743724 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:22.743731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:22.744063 (+ 332us) server_negotiation.cc:407] Connection header received
0809 20:29:22.744300 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:22.744304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:22.744349 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:22.744466 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:22.745392 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:22.745404 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:22.745408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:22.745422 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:22.745483 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:22.745504 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:22.745633 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:22.745757 (+ 124us) server_negotiation.cc:299] Negotiation successful
0809 20:29:22.745822 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 20:29:22.748373 3022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61: No bootstrap required, opened a new log
I20260809 20:29:22.755709 3022 raft_consensus.cc:348] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a263e88cbe3d41cea1bd2e65e04e6e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33659 } }
I20260809 20:29:22.756124 3022 raft_consensus.cc:374] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:22.756248 3022 raft_consensus.cc:729] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a263e88cbe3d41cea1bd2e65e04e6e61, State: Initialized, Role: FOLLOWER
I20260809 20:29:22.756852 3022 consensus_queue.cc:260] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a263e88cbe3d41cea1bd2e65e04e6e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33659 } }
I20260809 20:29:22.757110 3022 raft_consensus.cc:388] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:22.757283 3022 raft_consensus.cc:482] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:22.757498 3022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:22.759728 3022 raft_consensus.cc:504] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a263e88cbe3d41cea1bd2e65e04e6e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33659 } }
I20260809 20:29:22.760262 3022 leader_election.cc:302] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a263e88cbe3d41cea1bd2e65e04e6e61; no voters:
I20260809 20:29:22.760932 3022 leader_election.cc:288] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:22.761060 3032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:22.762316 3032 raft_consensus.cc:686] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [term 1 LEADER]: Becoming Leader. State: Replica: a263e88cbe3d41cea1bd2e65e04e6e61, State: Running, Role: LEADER
I20260809 20:29:22.762822 3032 consensus_queue.cc:237] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a263e88cbe3d41cea1bd2e65e04e6e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33659 } }
I20260809 20:29:22.763025 3022 sys_catalog.cc:564] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:22.765738 3034 sys_catalog.cc:455] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a263e88cbe3d41cea1bd2e65e04e6e61. Latest consensus state: current_term: 1 leader_uuid: "a263e88cbe3d41cea1bd2e65e04e6e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a263e88cbe3d41cea1bd2e65e04e6e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33659 } } }
I20260809 20:29:22.766214 3034 sys_catalog.cc:458] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:22.766855 3033 sys_catalog.cc:455] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a263e88cbe3d41cea1bd2e65e04e6e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a263e88cbe3d41cea1bd2e65e04e6e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33659 } } }
I20260809 20:29:22.767133 3033 sys_catalog.cc:458] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:22.768684 3042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:22.773902 3042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:22.784947 3042 catalog_manager.cc:1347] Generated new cluster ID: 391823a37fed4995a1b12ce76e4bd739
I20260809 20:29:22.785126 3042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:22.795269 3042 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:22.795770 3042 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:22.803757 3042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a263e88cbe3d41cea1bd2e65e04e6e61: Generated new TSK 0
I20260809 20:29:22.804749 3042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:22.817554 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2953
W20260809 20:29:22.818854 3016 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.60:51131
W20260809 20:29:22.819612 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 2953: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:29:22.819765 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 2953
2026-08-09T20:29:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2005 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:22 dist-test-slave-rnrw krb5kdc[3056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:22 dist-test-slave-rnrw krb5kdc[3056](info): set up 2 sockets
Aug 09 20:29:22 dist-test-slave-rnrw krb5kdc[3056](info): commencing operation
krb5kdc: starting...
W20260809 20:29:23.976480 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:24 dist-test-slave-rnrw krb5kdc[3056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:29:24.284162 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:45259
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:33461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:45259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:24.623884 3070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:24.624214 3070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:24.624384 3070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:24.631526 3070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:24.631700 3070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:24.631803 3070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:24.631914 3070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:24.632001 3070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:24.642452 3070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:33461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:45259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:45259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:24.643775 3070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:24.645165 3070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:24.657203 3076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:24.657271 3078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:24.657297 3075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:24.658115 3070 server_base.cc:1034] running on GCE node
I20260809 20:29:24.658847 3070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:24.660249 3070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:24.661448 3070 hybrid_clock.cc:648] HybridClock initialized: now 1786307364661410 us; error 35 us; skew 500 ppm
I20260809 20:29:24.661793 3070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:24 dist-test-slave-rnrw krb5kdc[3056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307364, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:24.669529 3070 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:29:24.671068 3070 webserver.cc:460] Webserver started at http://127.12.78.62:33171/ using document root <none> and password file <none>
I20260809 20:29:24.671761 3070 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:24.671895 3070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:24.672247 3070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:24.675307 3070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "f4356384d5174e4e800d6c0e763e0313"
format_stamp: "Formatted at 2026-08-09 20:29:24 on dist-test-slave-rnrw"
server_key: "f67c6e331f50e86d9a2d9bc3de960b62"
server_key_iv: "2b586e0b2434660e9fa714356ef82dd2"
server_key_version: "encryptionkey@0"
I20260809 20:29:24.676067 3070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "f4356384d5174e4e800d6c0e763e0313"
format_stamp: "Formatted at 2026-08-09 20:29:24 on dist-test-slave-rnrw"
server_key: "f67c6e331f50e86d9a2d9bc3de960b62"
server_key_iv: "2b586e0b2434660e9fa714356ef82dd2"
server_key_version: "encryptionkey@0"
I20260809 20:29:24.681208 3070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 20:29:24.684486 3085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:24.685788 3070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 20:29:24.685974 3070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "f4356384d5174e4e800d6c0e763e0313"
format_stamp: "Formatted at 2026-08-09 20:29:24 on dist-test-slave-rnrw"
server_key: "f67c6e331f50e86d9a2d9bc3de960b62"
server_key_iv: "2b586e0b2434660e9fa714356ef82dd2"
server_key_version: "encryptionkey@0"
I20260809 20:29:24.686173 3070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:24.707177 3070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:24.711691 3070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:24.712001 3070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:24.730839 3070 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:45259
I20260809 20:29:24.730877 3140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:45259 every 8 connection(s)
I20260809 20:29:24.732235 3070 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:24.733330 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 3070
I20260809 20:29:24.733522 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:24.734082 12600 external_mini_cluster.cc:1428] Setting key dc564419357ac247b007b1e9f4bc2148
I20260809 20:29:24.736398 3141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:24.746946 3141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313: Bootstrap starting.
Aug 09 20:29:24 dist-test-slave-rnrw krb5kdc[3056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:29:24.750108 3141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:24.751173 3141 log.cc:824] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:24.754423 3141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313: No bootstrap required, opened a new log
I20260809 20:29:24.756273 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:24.736154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51701 (local address 127.12.78.62:45259)
0809 20:29:24.736972 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:24.736983 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:24.737026 (+ 43us) server_negotiation.cc:407] Connection header received
0809 20:29:24.738651 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:24.738670 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:24.739041 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:24.739447 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:24.740984 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:24.742281 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:24.743514 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:24.744199 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:24.744422 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:24.748682 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:24.748691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:24.748704 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:24.748737 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:24.752060 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:24.752976 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:24.752983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:24.752988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:24.753163 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:24.753669 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:24.753672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:24.753674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:24.754180 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:24.754425 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:24.754959 (+ 534us) server_negotiation.cc:299] Negotiation successful
0809 20:29:24.755276 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":447,"threads_started":1}
I20260809 20:29:24.760363 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:24.763183 3141 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4356384d5174e4e800d6c0e763e0313" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 45259 } }
I20260809 20:29:24.763573 3141 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:24.763710 3141 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4356384d5174e4e800d6c0e763e0313, State: Initialized, Role: FOLLOWER
I20260809 20:29:24.764271 3141 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4356384d5174e4e800d6c0e763e0313" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 45259 } }
I20260809 20:29:24.764597 3141 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:24.764771 3141 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:24.765012 3141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:24.767410 3141 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4356384d5174e4e800d6c0e763e0313" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 45259 } }
I20260809 20:29:24.767939 3141 leader_election.cc:302] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4356384d5174e4e800d6c0e763e0313; no voters:
I20260809 20:29:24.768632 3141 leader_election.cc:288] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:24.768759 3151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:24.770327 3151 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [term 1 LEADER]: Becoming Leader. State: Replica: f4356384d5174e4e800d6c0e763e0313, State: Running, Role: LEADER
I20260809 20:29:24.771062 3141 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:24.770959 3151 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4356384d5174e4e800d6c0e763e0313" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 45259 } }
I20260809 20:29:24.774122 3153 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4356384d5174e4e800d6c0e763e0313. Latest consensus state: current_term: 1 leader_uuid: "f4356384d5174e4e800d6c0e763e0313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4356384d5174e4e800d6c0e763e0313" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 45259 } } }
I20260809 20:29:24.774710 3153 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:24.775417 3152 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4356384d5174e4e800d6c0e763e0313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4356384d5174e4e800d6c0e763e0313" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 45259 } } }
I20260809 20:29:24.777309 3152 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:24.782886 3158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:24.787602 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:24.764991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45593 (local address 127.12.78.62:45259)
0809 20:29:24.765186 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:24.765194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:24.765560 (+ 366us) server_negotiation.cc:407] Connection header received
0809 20:29:24.765778 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:24.765783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:24.765853 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:24.765965 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:24.767407 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:24.768338 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:24.770100 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:24.770807 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:24.770958 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:24.774138 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:24.774145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:24.774150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:24.774186 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:24.779871 (+ 5685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:24.783842 (+ 3971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:24.783853 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:24.783856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:24.784090 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:24.784869 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:24.784874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:24.784877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:24.785297 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:24.786789 (+ 1492us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:24.786983 (+ 194us) server_negotiation.cc:299] Negotiation successful
0809 20:29:24.787194 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 20:29:24.788448 3158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:24.799427 3158 catalog_manager.cc:1347] Generated new cluster ID: 315bd99b72bd498482ec1a560763ed8b
I20260809 20:29:24.799554 3158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:24.829612 3158 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:24.830927 3158 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:24.843127 3158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4356384d5174e4e800d6c0e763e0313: Generated new TSK 0
I20260809 20:29:24.843951 3158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:24.851329 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 3070
I20260809 20:29:24.852595 3136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:29:24.945966 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 3070
W20260809 20:29:24.965745 3054 connection.cc:535] client connection to 127.12.78.62:45259 recv error: Network error: failed to read from TLS socket (remote: 127.12.78.62:45259): Connection reset by peer (error 104)
2026-08-09T20:29:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2126 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:25 dist-test-slave-rnrw krb5kdc[3174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:25 dist-test-slave-rnrw krb5kdc[3174](info): set up 2 sockets
Aug 09 20:29:25 dist-test-slave-rnrw krb5kdc[3174](info): commencing operation
krb5kdc: starting...
W20260809 20:29:26.108384 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:26 dist-test-slave-rnrw krb5kdc[3174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:29:26.427594 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40073
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:37793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:40073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:26.785840 3188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:26.786196 3188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:26.786361 3188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:26.793931 3188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:26.794095 3188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:26.794214 3188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:26.794332 3188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:26.794417 3188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:26.804780 3188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:37793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:40073
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:26.806226 3188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:26.807638 3188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:26.819811 3197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:26.819820 3194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:26.820173 3193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:26.820612 3188 server_base.cc:1034] running on GCE node
I20260809 20:29:26.821424 3188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:26.822994 3188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:26.824231 3188 hybrid_clock.cc:648] HybridClock initialized: now 1786307366824201 us; error 42 us; skew 500 ppm
I20260809 20:29:26.824667 3188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:26 dist-test-slave-rnrw krb5kdc[3174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307366, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:26.827795 3188 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:29:26.829473 3188 webserver.cc:460] Webserver started at http://127.12.78.62:39541/ using document root <none> and password file <none>
I20260809 20:29:26.830183 3188 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:26.830338 3188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:26.830672 3188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:26.833798 3188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "b98218c6639241469ac20549292abc74"
format_stamp: "Formatted at 2026-08-09 20:29:26 on dist-test-slave-rnrw"
server_key: "6acac68acedfe18123aade19f824b446"
server_key_iv: "48a7af082efc82b466b2c8fe5b01aaa7"
server_key_version: "encryptionkey@0"
I20260809 20:29:26.834637 3188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "b98218c6639241469ac20549292abc74"
format_stamp: "Formatted at 2026-08-09 20:29:26 on dist-test-slave-rnrw"
server_key: "6acac68acedfe18123aade19f824b446"
server_key_iv: "48a7af082efc82b466b2c8fe5b01aaa7"
server_key_version: "encryptionkey@0"
I20260809 20:29:26.839907 3188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:29:26.843541 3204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:26.844933 3188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:29:26.845131 3188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "b98218c6639241469ac20549292abc74"
format_stamp: "Formatted at 2026-08-09 20:29:26 on dist-test-slave-rnrw"
server_key: "6acac68acedfe18123aade19f824b446"
server_key_iv: "48a7af082efc82b466b2c8fe5b01aaa7"
server_key_version: "encryptionkey@0"
I20260809 20:29:26.845340 3188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:26.873594 3188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:26.877830 3188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:26.878154 3188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:26.898218 3188 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:40073
I20260809 20:29:26.898277 3258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:40073 every 8 connection(s)
I20260809 20:29:26.899713 3188 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:26.903748 3259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:26.909533 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 3188
I20260809 20:29:26.909739 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:26.910310 12600 external_mini_cluster.cc:1428] Setting key 40e0eca0e4f5cbab0980f433d20e9e6c
I20260809 20:29:26.915248 3259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74: Bootstrap starting.
I20260809 20:29:26.919124 3259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:26.920394 3259 log.cc:824] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:26.923998 3259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74: No bootstrap required, opened a new log
I20260809 20:29:26.925339 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:26.912301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50827 (local address 127.12.78.62:40073)
0809 20:29:26.913236 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:26.913247 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:26.913298 (+ 51us) server_negotiation.cc:407] Connection header received
0809 20:29:26.915007 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:26.915031 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:26.915484 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:26.915905 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:26.916795 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:26.918202 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:26.920118 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:26.920945 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:26.921201 (+ 256us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:26.922383 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:26.922396 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:26.922414 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:26.923187 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:26.923289 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:26.923322 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:26.923490 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:26.923820 (+ 330us) server_negotiation.cc:299] Negotiation successful
0809 20:29:26.924004 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":488,"threads_started":1}
I20260809 20:29:26.929569 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:26.932683 3259 raft_consensus.cc:348] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98218c6639241469ac20549292abc74" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40073 } }
I20260809 20:29:26.933072 3259 raft_consensus.cc:374] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:26.933202 3259 raft_consensus.cc:729] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98218c6639241469ac20549292abc74, State: Initialized, Role: FOLLOWER
I20260809 20:29:26.933823 3259 consensus_queue.cc:260] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98218c6639241469ac20549292abc74" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40073 } }
I20260809 20:29:26.934130 3259 raft_consensus.cc:388] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:26.934284 3259 raft_consensus.cc:482] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:26.934592 3259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:26.937776 3259 raft_consensus.cc:504] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98218c6639241469ac20549292abc74" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40073 } }
I20260809 20:29:26.938361 3259 leader_election.cc:302] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b98218c6639241469ac20549292abc74; no voters:
I20260809 20:29:26.939108 3259 leader_election.cc:288] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:26.939273 3269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:26.940939 3269 raft_consensus.cc:686] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [term 1 LEADER]: Becoming Leader. State: Replica: b98218c6639241469ac20549292abc74, State: Running, Role: LEADER
I20260809 20:29:26.941597 3259 sys_catalog.cc:564] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:26.941624 3269 consensus_queue.cc:237] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98218c6639241469ac20549292abc74" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40073 } }
I20260809 20:29:26.942070 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:26.934222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56895 (local address 127.12.78.62:40073)
0809 20:29:26.934456 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:26.934465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:26.934968 (+ 503us) server_negotiation.cc:407] Connection header received
0809 20:29:26.935193 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:26.935197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:26.935267 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:26.935361 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:26.935986 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:26.936826 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:26.938699 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:26.939447 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:26.940462 (+ 1015us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:26.941132 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:26.941139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:26.941166 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:26.941188 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:26.941262 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:26.941284 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:26.941403 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:26.941658 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 20:29:26.941718 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 20:29:26.948805 3270 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b98218c6639241469ac20549292abc74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98218c6639241469ac20549292abc74" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40073 } } }
I20260809 20:29:26.949747 3270 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:26.949338 3271 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b98218c6639241469ac20549292abc74. Latest consensus state: current_term: 1 leader_uuid: "b98218c6639241469ac20549292abc74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98218c6639241469ac20549292abc74" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40073 } } }
I20260809 20:29:26.950160 3271 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74 [sys.catalog]: This master's current role is: LEADER
W20260809 20:29:26.953074 3285 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 20:29:26.953194 3285 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 20:29:26.953333 3286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:26.955515 3286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:26.963927 3286 catalog_manager.cc:1347] Generated new cluster ID: cd10063cea6f44aabec2ef7ad47542f5
I20260809 20:29:26.964043 3286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:26.984200 3286 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:26.986174 3286 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:26.998333 3286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b98218c6639241469ac20549292abc74: Generated new TSK 0
I20260809 20:29:26.999492 3286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:27.009088 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 3188
W20260809 20:29:27.010391 3254 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50827
W20260809 20:29:27.011215 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 3188: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:29:27.011381 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 3188
W20260809 20:29:27.031020 3172 connection.cc:535] client connection to 127.12.78.62:40073 recv error: Network error: failed to read from TLS socket (remote: 127.12.78.62:40073): Connection reset by peer (error 104)
2026-08-09T20:29:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2064 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:27 dist-test-slave-rnrw krb5kdc[3292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:27 dist-test-slave-rnrw krb5kdc[3292](info): set up 2 sockets
Aug 09 20:29:27 dist-test-slave-rnrw krb5kdc[3292](info): commencing operation
krb5kdc: starting...
W20260809 20:29:28.188704 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:28 dist-test-slave-rnrw krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:29:28.500164 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:33847
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:33847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:28.843384 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:28.843724 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:28.843907 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:28.851336 3305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:28.851454 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:28.851545 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:28.851629 3305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:28.851699 3305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:28.861734 3305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:33847
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:33847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:28.863101 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:28.864468 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:28.876730 3310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:28.876797 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:28.876824 3311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:28.878249 3305 server_base.cc:1034] running on GCE node
I20260809 20:29:28.879086 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:28.880630 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:28.881853 3305 hybrid_clock.cc:648] HybridClock initialized: now 1786307368881793 us; error 54 us; skew 500 ppm
I20260809 20:29:28.882222 3305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:28 dist-test-slave-rnrw krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307368, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:28.885496 3305 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:29:28.887140 3305 webserver.cc:460] Webserver started at http://127.12.78.62:46351/ using document root <none> and password file <none>
I20260809 20:29:28.887887 3305 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:28.888042 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:28.888372 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:28.891527 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "54937ca981ed410a93af9deade2e6089"
format_stamp: "Formatted at 2026-08-09 20:29:28 on dist-test-slave-rnrw"
server_key: "c0502b6b6778eea46da4c99869d08d51"
server_key_iv: "43f8e0ba8ac52b3d58a17d9f3a8634a7"
server_key_version: "encryptionkey@0"
I20260809 20:29:28.892343 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "54937ca981ed410a93af9deade2e6089"
format_stamp: "Formatted at 2026-08-09 20:29:28 on dist-test-slave-rnrw"
server_key: "c0502b6b6778eea46da4c99869d08d51"
server_key_iv: "43f8e0ba8ac52b3d58a17d9f3a8634a7"
server_key_version: "encryptionkey@0"
I20260809 20:29:28.897768 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 20:29:28.901427 3320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:28.902769 3305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:29:28.902966 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "54937ca981ed410a93af9deade2e6089"
format_stamp: "Formatted at 2026-08-09 20:29:28 on dist-test-slave-rnrw"
server_key: "c0502b6b6778eea46da4c99869d08d51"
server_key_iv: "43f8e0ba8ac52b3d58a17d9f3a8634a7"
server_key_version: "encryptionkey@0"
I20260809 20:29:28.903177 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:28.909333 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:28.913426 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:28.913749 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:28.933346 3305 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:33847
I20260809 20:29:28.933388 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:33847 every 8 connection(s)
I20260809 20:29:28.934928 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:28.939065 3376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:28.940016 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 3305
I20260809 20:29:28.940224 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:28.940861 12600 external_mini_cluster.cc:1428] Setting key ea7a01414d52c48e478ee3b243faa77b
I20260809 20:29:28.949687 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:28.942911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56009 (local address 127.12.78.62:33847)
0809 20:29:28.943789 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:28.943800 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:28.943852 (+ 52us) server_negotiation.cc:407] Connection header received
0809 20:29:28.946194 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:28.946220 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:28.946565 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:28.946982 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:28.948054 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:28.948061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:28.948075 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:28.948090 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:28.948156 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:28.948180 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:28.948437 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:28.948592 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 20:29:28.948775 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":359,"threads_started":1}
I20260809 20:29:28.954018 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:28.954782 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089: Bootstrap starting.
I20260809 20:29:28.958326 3376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:28.959839 3376 log.cc:824] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:28.961485 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:28.958712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59223 (local address 127.12.78.62:33847)
0809 20:29:28.958952 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:28.958959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:28.959404 (+ 445us) server_negotiation.cc:407] Connection header received
0809 20:29:28.959549 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:28.959557 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:28.959641 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:28.959767 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:28.960770 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:28.960774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:28.960781 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:28.960796 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:28.960838 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:28.960856 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:28.960984 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:28.961106 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 20:29:28.961184 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 20:29:28.963235 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089: No bootstrap required, opened a new log
I20260809 20:29:28.970849 3376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54937ca981ed410a93af9deade2e6089" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33847 } }
I20260809 20:29:28.971208 3376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:28.971292 3376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54937ca981ed410a93af9deade2e6089, State: Initialized, Role: FOLLOWER
I20260809 20:29:28.971932 3376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54937ca981ed410a93af9deade2e6089" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33847 } }
I20260809 20:29:28.972198 3376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:28.972371 3376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:28.972584 3376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:28.974786 3376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54937ca981ed410a93af9deade2e6089" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33847 } }
I20260809 20:29:28.975245 3376 leader_election.cc:302] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54937ca981ed410a93af9deade2e6089; no voters:
I20260809 20:29:28.975837 3376 leader_election.cc:288] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:28.975989 3386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:28.976720 3386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [term 1 LEADER]: Becoming Leader. State: Replica: 54937ca981ed410a93af9deade2e6089, State: Running, Role: LEADER
I20260809 20:29:28.977236 3386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54937ca981ed410a93af9deade2e6089" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33847 } }
I20260809 20:29:28.978014 3376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:28.984674 3387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54937ca981ed410a93af9deade2e6089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54937ca981ed410a93af9deade2e6089" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33847 } } }
I20260809 20:29:28.984674 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54937ca981ed410a93af9deade2e6089. Latest consensus state: current_term: 1 leader_uuid: "54937ca981ed410a93af9deade2e6089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54937ca981ed410a93af9deade2e6089" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 33847 } } }
I20260809 20:29:28.985356 3387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:28.985438 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:28.987658 3393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:28.993428 3393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:29.003268 3393 catalog_manager.cc:1347] Generated new cluster ID: a0e3b09cc04e48b7b263007f423b6a6c
I20260809 20:29:29.003418 3393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:29.023453 3393 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:29.023965 3393 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:29.044724 3393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54937ca981ed410a93af9deade2e6089: Generated new TSK 0
I20260809 20:29:29.045637 3393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:29.058255 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 3305
W20260809 20:29:29.059409 3371 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56009
W20260809 20:29:29.060108 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 3305: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:29:29.060258 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 3305
2026-08-09T20:29:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2049 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:29 dist-test-slave-rnrw krb5kdc[3407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:29 dist-test-slave-rnrw krb5kdc[3407](info): set up 2 sockets
Aug 09 20:29:29 dist-test-slave-rnrw krb5kdc[3407](info): commencing operation
krb5kdc: starting...
W20260809 20:29:30.517236 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.418s user 0.010s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:30 dist-test-slave-rnrw krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:29:30.813838 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:35907
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:45089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:35907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:31.152992 3420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:31.153352 3420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:31.153524 3420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:31.160735 3420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:31.160900 3420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:31.161011 3420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:31.161124 3420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:31.161214 3420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:31.171806 3420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:45089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:35907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:35907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:31.173254 3420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:31.174633 3420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:31.186872 3425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:31.186816 3429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:31.186839 3426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:31.187615 3420 server_base.cc:1034] running on GCE node
I20260809 20:29:31.188376 3420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:31.189824 3420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:31.191023 3420 hybrid_clock.cc:648] HybridClock initialized: now 1786307371190965 us; error 51 us; skew 500 ppm
I20260809 20:29:31.191370 3420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:31 dist-test-slave-rnrw krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307371, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:31.199743 3420 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:29:31.201376 3420 webserver.cc:460] Webserver started at http://127.12.78.62:37763/ using document root <none> and password file <none>
I20260809 20:29:31.202028 3420 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:31.202131 3420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:31.202399 3420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:31.205541 3420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "13fb36aa3cbc448199e197459dd63de1"
format_stamp: "Formatted at 2026-08-09 20:29:31 on dist-test-slave-rnrw"
server_key: "0a3d9582e77f698671da9872cecf588f"
server_key_iv: "15880f5c8f197dbbf90b6a198689ace5"
server_key_version: "encryptionkey@0"
I20260809 20:29:31.206282 3420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "13fb36aa3cbc448199e197459dd63de1"
format_stamp: "Formatted at 2026-08-09 20:29:31 on dist-test-slave-rnrw"
server_key: "0a3d9582e77f698671da9872cecf588f"
server_key_iv: "15880f5c8f197dbbf90b6a198689ace5"
server_key_version: "encryptionkey@0"
I20260809 20:29:31.211575 3420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:29:31.215034 3436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:31.216490 3420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 20:29:31.216707 3420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "13fb36aa3cbc448199e197459dd63de1"
format_stamp: "Formatted at 2026-08-09 20:29:31 on dist-test-slave-rnrw"
server_key: "0a3d9582e77f698671da9872cecf588f"
server_key_iv: "15880f5c8f197dbbf90b6a198689ace5"
server_key_version: "encryptionkey@0"
I20260809 20:29:31.216917 3420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:31.233124 3420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:31.237468 3420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:31.237782 3420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:31.257289 3420 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:35907
I20260809 20:29:31.257324 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:35907 every 8 connection(s)
I20260809 20:29:31.258747 3420 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:31.262796 3495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:31.263432 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 3420
I20260809 20:29:31.263623 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:31.264063 12600 external_mini_cluster.cc:1428] Setting key 2017bfa8cd5543ac5bf0b258e4e572a5
I20260809 20:29:31.277936 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1: Bootstrap starting.
Aug 09 20:29:31 dist-test-slave-rnrw krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:29:31.281220 3495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:31.282660 3495 log.cc:824] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:31.286084 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1: No bootstrap required, opened a new log
I20260809 20:29:31.289677 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:31.265922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58095 (local address 127.12.78.62:35907)
0809 20:29:31.266775 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:31.266792 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:31.266843 (+ 51us) server_negotiation.cc:407] Connection header received
0809 20:29:31.269882 (+ 3039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:31.269912 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:31.270464 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:31.270927 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:31.272305 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:31.274359 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:31.275624 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:31.276704 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:31.276900 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:31.281010 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:31.281024 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:31.281044 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:31.281102 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:31.284995 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:31.285700 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:31.285710 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:31.285718 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:31.285928 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:31.286346 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:31.286351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:31.286354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:31.287092 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:31.287412 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:31.287740 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 20:29:31.288207 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":357,"threads_started":1}
I20260809 20:29:31.293475 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:31.295662 3495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13fb36aa3cbc448199e197459dd63de1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35907 } }
I20260809 20:29:31.296036 3495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:31.296160 3495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13fb36aa3cbc448199e197459dd63de1, State: Initialized, Role: FOLLOWER
I20260809 20:29:31.296842 3495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13fb36aa3cbc448199e197459dd63de1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35907 } }
I20260809 20:29:31.297101 3495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:31.297251 3495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:31.297426 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:31.300966 3495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13fb36aa3cbc448199e197459dd63de1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35907 } }
I20260809 20:29:31.301496 3495 leader_election.cc:302] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13fb36aa3cbc448199e197459dd63de1; no voters:
I20260809 20:29:31.302176 3495 leader_election.cc:288] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:31.302284 3506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:31.304322 3495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:31.304831 3506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [term 1 LEADER]: Becoming Leader. State: Replica: 13fb36aa3cbc448199e197459dd63de1, State: Running, Role: LEADER
I20260809 20:29:31.305444 3506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13fb36aa3cbc448199e197459dd63de1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35907 } }
I20260809 20:29:31.309212 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13fb36aa3cbc448199e197459dd63de1. Latest consensus state: current_term: 1 leader_uuid: "13fb36aa3cbc448199e197459dd63de1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13fb36aa3cbc448199e197459dd63de1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35907 } } }
I20260809 20:29:31.309410 3507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13fb36aa3cbc448199e197459dd63de1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13fb36aa3cbc448199e197459dd63de1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 35907 } } }
I20260809 20:29:31.309747 3507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:31.309758 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:31.313231 3513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:31.316781 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:31.297940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45457 (local address 127.12.78.62:35907)
0809 20:29:31.298183 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:31.298192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:31.298691 (+ 499us) server_negotiation.cc:407] Connection header received
0809 20:29:31.298861 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:31.298865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:31.298929 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:31.299012 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:31.300871 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:31.301714 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:31.303619 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:31.304680 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:31.304849 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:31.308178 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:31.308183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:31.308186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:31.308216 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:31.312853 (+ 4637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:31.313791 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:31.313797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:31.313799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:31.313945 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:31.314415 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:31.314420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:31.314423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:31.314868 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:31.315131 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:31.315928 (+ 797us) server_negotiation.cc:299] Negotiation successful
0809 20:29:31.316234 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 20:29:31.323164 3513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:31.333041 3513 catalog_manager.cc:1347] Generated new cluster ID: 4046e2cdee5c4281b46683f216df9892
I20260809 20:29:31.333197 3513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:31.354157 3513 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:31.355489 3513 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:31.370283 3513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13fb36aa3cbc448199e197459dd63de1: Generated new TSK 0
I20260809 20:29:31.371403 3513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:31.384275 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 3420
I20260809 20:29:31.385526 3489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:29:31.481721 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 3420
2026-08-09T20:29:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:31 dist-test-slave-rnrw krb5kdc[3529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:31 dist-test-slave-rnrw krb5kdc[3529](info): set up 2 sockets
Aug 09 20:29:31 dist-test-slave-rnrw krb5kdc[3529](info): commencing operation
krb5kdc: starting...
W20260809 20:29:32.732039 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.001s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:32 dist-test-slave-rnrw krb5kdc[3529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:29:33.059580 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:34079
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:34079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:33.406808 3542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:33.407150 3542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:33.407325 3542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:33.414649 3542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:33.414803 3542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:33.414906 3542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:33.415014 3542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:33.415102 3542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:33.425292 3542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:34079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:34079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:33.426695 3542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:33.428107 3542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:33.440886 3547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:33.440862 3550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:33.441072 3548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:33.441769 3542 server_base.cc:1034] running on GCE node
I20260809 20:29:33.442625 3542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:33.444120 3542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:33.445353 3542 hybrid_clock.cc:648] HybridClock initialized: now 1786307373445307 us; error 58 us; skew 500 ppm
I20260809 20:29:33.445721 3542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:33 dist-test-slave-rnrw krb5kdc[3529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307373, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:33.448957 3542 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:29:33.450440 3542 webserver.cc:460] Webserver started at http://127.12.78.62:38187/ using document root <none> and password file <none>
I20260809 20:29:33.451153 3542 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:33.451308 3542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:33.451618 3542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:33.454682 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "512c9ee7041a417cb0f3c294524c6966"
format_stamp: "Formatted at 2026-08-09 20:29:33 on dist-test-slave-rnrw"
server_key: "2c2917f086c6698525b8d48802c413ef"
server_key_iv: "29116b770db4f920dcc4965c50cdb8e4"
server_key_version: "encryptionkey@0"
I20260809 20:29:33.455471 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "512c9ee7041a417cb0f3c294524c6966"
format_stamp: "Formatted at 2026-08-09 20:29:33 on dist-test-slave-rnrw"
server_key: "2c2917f086c6698525b8d48802c413ef"
server_key_iv: "29116b770db4f920dcc4965c50cdb8e4"
server_key_version: "encryptionkey@0"
I20260809 20:29:33.460772 3542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 20:29:33.464141 3557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:33.465510 3542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:29:33.465711 3542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "512c9ee7041a417cb0f3c294524c6966"
format_stamp: "Formatted at 2026-08-09 20:29:33 on dist-test-slave-rnrw"
server_key: "2c2917f086c6698525b8d48802c413ef"
server_key_iv: "29116b770db4f920dcc4965c50cdb8e4"
server_key_version: "encryptionkey@0"
I20260809 20:29:33.465914 3542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:33.501498 3542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:33.506340 3542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:33.506661 3542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:33.525602 3542 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:34079
I20260809 20:29:33.525640 3610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:34079 every 8 connection(s)
I20260809 20:29:33.527109 3542 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:33.530431 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 3542
I20260809 20:29:33.530616 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:33.531220 12600 external_mini_cluster.cc:1428] Setting key 06033ddaacec43af0f92fea228ee39c5
I20260809 20:29:33.531318 3611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:33.544991 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:33.533091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33525 (local address 127.12.78.62:34079)
0809 20:29:33.533906 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:33.533917 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:33.533961 (+ 44us) server_negotiation.cc:407] Connection header received
0809 20:29:33.535958 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:33.535980 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:33.536402 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:33.536831 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:33.537434 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:33.538778 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:33.540592 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:33.541289 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:33.541421 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:33.542687 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:33.542695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:33.542708 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:33.543264 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:33.543339 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:33.543363 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:33.543450 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:33.543683 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 20:29:33.543944 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":598,"thread_start_us":365,"threads_started":1}
I20260809 20:29:33.545423 3611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966: Bootstrap starting.
I20260809 20:29:33.548647 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:33.548950 3611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:33.549913 3611 log.cc:824] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:33.552817 3611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966: No bootstrap required, opened a new log
I20260809 20:29:33.559554 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:33.553497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35129 (local address 127.12.78.62:34079)
0809 20:29:33.553697 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:33.553705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:33.554084 (+ 379us) server_negotiation.cc:407] Connection header received
0809 20:29:33.554299 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:33.554303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:33.554360 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:33.554463 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:33.555018 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:33.555768 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:33.557125 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:33.557618 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:33.557770 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:33.558820 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:33.558826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:33.558829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:33.558846 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:33.558880 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:33.558894 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:33.559003 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:33.559160 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 20:29:33.559251 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 20:29:33.561432 3611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512c9ee7041a417cb0f3c294524c6966" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 34079 } }
I20260809 20:29:33.561810 3611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:33.561916 3611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512c9ee7041a417cb0f3c294524c6966, State: Initialized, Role: FOLLOWER
I20260809 20:29:33.562462 3611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512c9ee7041a417cb0f3c294524c6966" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 34079 } }
I20260809 20:29:33.562702 3611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:33.562848 3611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:33.563031 3611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:33.565685 3611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512c9ee7041a417cb0f3c294524c6966" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 34079 } }
I20260809 20:29:33.566179 3611 leader_election.cc:302] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 512c9ee7041a417cb0f3c294524c6966; no voters:
I20260809 20:29:33.566764 3611 leader_election.cc:288] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:33.566911 3622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:33.567600 3622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [term 1 LEADER]: Becoming Leader. State: Replica: 512c9ee7041a417cb0f3c294524c6966, State: Running, Role: LEADER
I20260809 20:29:33.568100 3622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512c9ee7041a417cb0f3c294524c6966" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 34079 } }
I20260809 20:29:33.568888 3611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:33.570884 3623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "512c9ee7041a417cb0f3c294524c6966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512c9ee7041a417cb0f3c294524c6966" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 34079 } } }
I20260809 20:29:33.571286 3623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:33.571779 3624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 512c9ee7041a417cb0f3c294524c6966. Latest consensus state: current_term: 1 leader_uuid: "512c9ee7041a417cb0f3c294524c6966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512c9ee7041a417cb0f3c294524c6966" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 34079 } } }
I20260809 20:29:33.572032 3624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:33.574971 3628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:33.580773 3628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:33.589828 3628 catalog_manager.cc:1347] Generated new cluster ID: 8c90f1f6f9944d489324e09894f4cbbc
I20260809 20:29:33.589942 3628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:33.602180 3628 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:33.603511 3628 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:33.613831 3628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 512c9ee7041a417cb0f3c294524c6966: Generated new TSK 0
I20260809 20:29:33.614653 3628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:33.624603 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 3542
W20260809 20:29:33.625610 3605 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33525
W20260809 20:29:33.626276 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 3542: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:29:33.626418 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 3542
2026-08-09T20:29:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2142 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:33 dist-test-slave-rnrw krb5kdc[3644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:33 dist-test-slave-rnrw krb5kdc[3644](info): set up 2 sockets
Aug 09 20:29:33 dist-test-slave-rnrw krb5kdc[3644](info): commencing operation
krb5kdc: starting...
W20260809 20:29:34.810559 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.001s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:35 dist-test-slave-rnrw krb5kdc[3644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:29:35.131947 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:46735
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:46735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:35.501791 3658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:35.502086 3658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:35.502238 3658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:35.509918 3658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:35.510049 3658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:35.510185 3658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:35.510295 3658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:35.510380 3658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:35.520792 3658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:46735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:46735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:35.522150 3658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:35.523496 3658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:35.535864 3664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:35.536003 3665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:35.536046 3667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:35.537184 3658 server_base.cc:1034] running on GCE node
I20260809 20:29:35.538074 3658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:35.539609 3658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:35.540831 3658 hybrid_clock.cc:648] HybridClock initialized: now 1786307375540781 us; error 38 us; skew 500 ppm
I20260809 20:29:35.541229 3658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:35 dist-test-slave-rnrw krb5kdc[3644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307375, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:35.544898 3658 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:29:35.546597 3658 webserver.cc:460] Webserver started at http://127.12.78.62:42191/ using document root <none> and password file <none>
I20260809 20:29:35.547295 3658 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:35.547406 3658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:35.547715 3658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:35.550959 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "a0b365229a1f47c88dde399a79fa5a65"
format_stamp: "Formatted at 2026-08-09 20:29:35 on dist-test-slave-rnrw"
server_key: "02ed4816d28b85e9e04226e148754b17"
server_key_iv: "ffaaf0ae323c0f6c04c3b5237d05c183"
server_key_version: "encryptionkey@0"
I20260809 20:29:35.551707 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "a0b365229a1f47c88dde399a79fa5a65"
format_stamp: "Formatted at 2026-08-09 20:29:35 on dist-test-slave-rnrw"
server_key: "02ed4816d28b85e9e04226e148754b17"
server_key_iv: "ffaaf0ae323c0f6c04c3b5237d05c183"
server_key_version: "encryptionkey@0"
I20260809 20:29:35.557252 3658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 20:29:35.560941 3674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:35.562325 3658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 20:29:35.562525 3658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "a0b365229a1f47c88dde399a79fa5a65"
format_stamp: "Formatted at 2026-08-09 20:29:35 on dist-test-slave-rnrw"
server_key: "02ed4816d28b85e9e04226e148754b17"
server_key_iv: "ffaaf0ae323c0f6c04c3b5237d05c183"
server_key_version: "encryptionkey@0"
I20260809 20:29:35.562750 3658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:35.568830 3658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:35.572921 3658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:35.573239 3658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:35.592518 3658 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:46735
I20260809 20:29:35.592608 3727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:46735 every 8 connection(s)
I20260809 20:29:35.594020 3658 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:35.597746 3728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:35.603255 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 3658
I20260809 20:29:35.603542 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:35.603997 12600 external_mini_cluster.cc:1428] Setting key 28c7623cf8a1afc3ca680ccb625f613d
I20260809 20:29:35.609395 3728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65: Bootstrap starting.
I20260809 20:29:35.613128 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:35.605957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49853 (local address 127.12.78.62:46735)
0809 20:29:35.606875 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:35.606890 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:35.606944 (+ 54us) server_negotiation.cc:407] Connection header received
0809 20:29:35.609337 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:35.609365 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:35.609785 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:35.610199 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:35.611188 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:35.611197 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:35.611220 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:35.611241 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:35.611313 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:35.611341 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:35.611571 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:35.611707 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 20:29:35.611907 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":387,"threads_started":1}
I20260809 20:29:35.613492 3728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:35.614557 3728 log.cc:824] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:35.616972 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:35.617532 3728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65: No bootstrap required, opened a new log
I20260809 20:29:35.627398 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:35.621386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34777 (local address 127.12.78.62:46735)
0809 20:29:35.621545 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:35.621549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:35.625403 (+ 3854us) server_negotiation.cc:407] Connection header received
0809 20:29:35.625611 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:35.625616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:35.625665 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:35.625792 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:35.626717 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:35.626722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:35.626727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 20:29:35.626745 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:35.626791 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 20:29:35.626806 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:35.626936 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:35.627029 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 20:29:35.627089 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 20:29:35.627242 3728 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b365229a1f47c88dde399a79fa5a65" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46735 } }
I20260809 20:29:35.627806 3728 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 20:29:35.627931 3728 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b365229a1f47c88dde399a79fa5a65, State: Initialized, Role: FOLLOWER
I20260809 20:29:35.628625 3728 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b365229a1f47c88dde399a79fa5a65" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46735 } }
I20260809 20:29:35.628944 3728 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 20:29:35.629097 3728 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 20:29:35.629295 3728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:35.631754 3728 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b365229a1f47c88dde399a79fa5a65" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46735 } }
I20260809 20:29:35.632297 3728 leader_election.cc:302] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0b365229a1f47c88dde399a79fa5a65; no voters:
I20260809 20:29:35.632980 3728 leader_election.cc:288] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 20:29:35.633158 3738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:35.633894 3738 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [term 1 LEADER]: Becoming Leader. State: Replica: a0b365229a1f47c88dde399a79fa5a65, State: Running, Role: LEADER
I20260809 20:29:35.634554 3738 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b365229a1f47c88dde399a79fa5a65" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46735 } }
I20260809 20:29:35.635286 3728 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:35.638495 3739 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0b365229a1f47c88dde399a79fa5a65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b365229a1f47c88dde399a79fa5a65" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46735 } } }
I20260809 20:29:35.638460 3740 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0b365229a1f47c88dde399a79fa5a65. Latest consensus state: current_term: 1 leader_uuid: "a0b365229a1f47c88dde399a79fa5a65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b365229a1f47c88dde399a79fa5a65" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 46735 } } }
I20260809 20:29:35.638937 3739 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:35.638965 3740 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:35.640518 3746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:35.646006 3746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:35.655460 3746 catalog_manager.cc:1347] Generated new cluster ID: 5e4ace54852e4c468327d2d01268aba2
I20260809 20:29:35.655584 3746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:35.667160 3746 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:35.667724 3746 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:35.684506 3746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0b365229a1f47c88dde399a79fa5a65: Generated new TSK 0
I20260809 20:29:35.685729 3746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:35.694581 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 3658
W20260809 20:29:35.695712 3722 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49853
W20260809 20:29:35.696411 12600 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 3658: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 20:29:35.696622 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 3658
2026-08-09T20:29:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2069 ms)
[----------] 18 tests from AuthTokenIssuingTest (37886 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:35 dist-test-slave-rnrw krb5kdc[3759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:35 dist-test-slave-rnrw krb5kdc[3759](info): set up 2 sockets
Aug 09 20:29:35 dist-test-slave-rnrw krb5kdc[3759](info): commencing operation
krb5kdc: starting...
W20260809 20:29:36.852468 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.006s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:37 dist-test-slave-rnrw krb5kdc[3759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:29:37.165210 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40065
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:40065,127.12.78.61:37059,127.12.78.60:41465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:37.521205 3772 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 20:29:37.521615 3772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:37.521731 3772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:37.521888 3772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:37.529301 3772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:37.529400 3772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:37.529538 3772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:37.529633 3772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:37.529731 3772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:37.539917 3772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:40065,127.12.78.61:37059,127.12.78.60:41465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:40065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:37.541349 3772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:37.542809 3772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:37.555308 3778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:37.555308 3781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:37.555555 3777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:37.556908 3772 server_base.cc:1034] running on GCE node
I20260809 20:29:37.557943 3772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:37.559576 3772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:37.560871 3772 hybrid_clock.cc:648] HybridClock initialized: now 1786307377560790 us; error 51 us; skew 500 ppm
I20260809 20:29:37.561282 3772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:37 dist-test-slave-rnrw krb5kdc[3759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307377, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:37.568692 3772 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:29:37.570513 3772 webserver.cc:460] Webserver started at http://127.12.78.62:36517/ using document root <none> and password file <none>
I20260809 20:29:37.571306 3772 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:37.571455 3772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:37.571871 3772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:37.575188 3772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "97cc7bfd127146739fb4b1f167c0cec1"
format_stamp: "Formatted at 2026-08-09 20:29:37 on dist-test-slave-rnrw"
server_key: "71617e3c3652a03e814fd40ce7289498"
server_key_iv: "26263e651f0f4210debefda536390e67"
server_key_version: "encryptionkey@0"
I20260809 20:29:37.576069 3772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "97cc7bfd127146739fb4b1f167c0cec1"
format_stamp: "Formatted at 2026-08-09 20:29:37 on dist-test-slave-rnrw"
server_key: "71617e3c3652a03e814fd40ce7289498"
server_key_iv: "26263e651f0f4210debefda536390e67"
server_key_version: "encryptionkey@0"
I20260809 20:29:37.581665 3772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 20:29:37.585253 3789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:37.586541 3772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 20:29:37.586737 3772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "97cc7bfd127146739fb4b1f167c0cec1"
format_stamp: "Formatted at 2026-08-09 20:29:37 on dist-test-slave-rnrw"
server_key: "71617e3c3652a03e814fd40ce7289498"
server_key_iv: "26263e651f0f4210debefda536390e67"
server_key_version: "encryptionkey@0"
I20260809 20:29:37.586966 3772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:37.603462 3772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:37.607491 3772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:37.607771 3772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:37.627429 3772 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:40065
I20260809 20:29:37.627478 3843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:40065 every 8 connection(s)
I20260809 20:29:37.628810 3772 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:37.632447 3844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:37.637409 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 3772
I20260809 20:29:37.637657 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:37.638520 12600 external_mini_cluster.cc:1428] Setting key 5b4b54161c788a14ab65fe26cd02beb2
I20260809 20:29:37.640074 3844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:29:37 dist-test-slave-rnrw krb5kdc[3759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307377, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:29:37.664639 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:37.644111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40065 (local address 127.0.0.1:51104)
0809 20:29:37.645488 (+ 1377us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:37.645541 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:37.647341 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:37.648303 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:37.648308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:37.649098 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:37.649442 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:37.649459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:37.651764 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:37.651770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:37.652924 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:37.652936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:37.653856 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:37.653862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:37.654040 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:37.654725 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:37.654747 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:37.656992 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:37.660189 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:37.660200 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:37.660212 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:37.660698 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:37.661109 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:37.661114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:37.661118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:37.661298 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:37.662059 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:37.662067 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:37.662294 (+ 227us) client_negotiation.cc:769] Sending connection context
0809 20:29:37.662555 (+ 261us) client_negotiation.cc:240] Negotiation successful
0809 20:29:37.663225 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"mutex_wait_us":171,"spinlock_wait_cycles":15104,"thread_start_us":879,"threads_started":1}
I20260809 20:29:37.664657 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:37.644111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.12.78.62:40065)
0809 20:29:37.645103 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:37.645118 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:37.645683 (+ 565us) server_negotiation.cc:407] Connection header received
0809 20:29:37.647756 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:37.647766 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:37.648062 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:37.648290 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:37.649671 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:37.651589 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:37.653063 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:37.653608 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:37.653833 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:37.657233 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:37.657241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:37.657253 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:37.657285 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:37.660018 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:37.660836 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:37.660843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:37.660846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:37.660998 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:37.661416 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:37.661425 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:37.661427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:37.661913 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:37.662134 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:37.662697 (+ 563us) server_negotiation.cc:299] Negotiation successful
0809 20:29:37.663107 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":723,"thread_start_us":491,"threads_started":1}
I20260809 20:29:37.667706 3844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.78.61@KRBTEST.COM; defaulting to no policy
I20260809 20:29:37.668877 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:37.668437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:37059 (local address 127.0.0.1:56310)
0809 20:29:37.668657 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:37.668765 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260809 20:29:37.669126 3791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
I20260809 20:29:37.670812 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:37.670413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:37059 (local address 127.0.0.1:56320)
0809 20:29:37.670593 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:37.670684 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 20:29:37.671085 3844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:37059: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.61.keytab.
Entry for principal kudu/127.12.78.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.61.keytab.
I20260809 20:29:37.705946 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.61:37059
--webserver_interface=127.12.78.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.61.keytab
--principal=kudu/127.12.78.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:40065,127.12.78.61:37059,127.12.78.60:41465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:29:37.720456 3844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } attempt: 1
I20260809 20:29:37.721755 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:37.721286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:37059 (local address 127.0.0.1:56324)
0809 20:29:37.721497 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:37.721621 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260809 20:29:37.722862 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:37.722498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:37059 (local address 127.0.0.1:56334)
0809 20:29:37.722668 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:37.722735 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 20:29:37.723083 3844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:37059: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
I20260809 20:29:37.791399 3844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } attempt: 2
I20260809 20:29:37.792728 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:37.792135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:37059 (local address 127.0.0.1:56350)
0809 20:29:37.792397 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:37.792486 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 20:29:37.793809 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:37.793438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:37059 (local address 127.0.0.1:56360)
0809 20:29:37.793608 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:37.793680 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 20:29:37.794031 3844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:37059: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
I20260809 20:29:37.885316 3844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } attempt: 3
I20260809 20:29:37.886463 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:37.886015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:37059 (local address 127.0.0.1:56372)
0809 20:29:37.886221 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:37.886320 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260809 20:29:37.887625 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:37.887234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:37059 (local address 127.0.0.1:56382)
0809 20:29:37.887399 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:37.887478 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 20:29:37.887828 3844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:37059: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
I20260809 20:29:38.031118 3844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } attempt: 4
I20260809 20:29:38.032346 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.031811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:37059 (local address 127.0.0.1:56390)
0809 20:29:38.032123 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.032210 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260809 20:29:38.033529 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.033118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:37059 (local address 127.0.0.1:56406)
0809 20:29:38.033313 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.033402 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260809 20:29:38.033821 3844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:37059: Network error: Client connection negotiation failed: client connection to 127.12.78.61:37059: connect: Connection refused (error 111)
W20260809 20:29:38.051842 3851 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 20:29:38.052204 3851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:38.052330 3851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:38.052469 3851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:38.060104 3851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:38.060212 3851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:38.060343 3851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:38.060420 3851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:38.060524 3851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:38.070771 3851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:40065,127.12.78.61:37059,127.12.78.60:41465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.61.keytab
--principal=kudu/127.12.78.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.61:37059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.78.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:38.072077 3851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:38.073477 3851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:38.085202 3859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:38.085184 3857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:38.085304 3856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:38.086086 3851 server_base.cc:1034] running on GCE node
I20260809 20:29:38.086781 3851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:38.088056 3851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:38.089283 3851 hybrid_clock.cc:648] HybridClock initialized: now 1786307378089203 us; error 45 us; skew 500 ppm
I20260809 20:29:38.089622 3851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:38 dist-test-slave-rnrw krb5kdc[3759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307378, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:38.093278 3851 init.cc:377] Logged in from keytab as kudu/127.12.78.61@KRBTEST.COM (short username kudu)
I20260809 20:29:38.094652 3851 webserver.cc:460] Webserver started at http://127.12.78.61:38115/ using document root <none> and password file <none>
I20260809 20:29:38.095309 3851 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:38.095444 3851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:38.095793 3851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:38.098863 3851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/data/instance:
uuid: "7a50a6848d894e6f81aa8dcef0a6c53f"
format_stamp: "Formatted at 2026-08-09 20:29:38 on dist-test-slave-rnrw"
server_key: "b2fc6d53887ab47abe2082b6ac3aea5b"
server_key_iv: "6c35c1f7397c068cce134652d740763b"
server_key_version: "encryptionkey@0"
I20260809 20:29:38.099682 3851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/wal/instance:
uuid: "7a50a6848d894e6f81aa8dcef0a6c53f"
format_stamp: "Formatted at 2026-08-09 20:29:38 on dist-test-slave-rnrw"
server_key: "b2fc6d53887ab47abe2082b6ac3aea5b"
server_key_iv: "6c35c1f7397c068cce134652d740763b"
server_key_version: "encryptionkey@0"
I20260809 20:29:38.105013 3851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 20:29:38.108465 3866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:38.109874 3851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 20:29:38.110062 3851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/wal
uuid: "7a50a6848d894e6f81aa8dcef0a6c53f"
format_stamp: "Formatted at 2026-08-09 20:29:38 on dist-test-slave-rnrw"
server_key: "b2fc6d53887ab47abe2082b6ac3aea5b"
server_key_iv: "6c35c1f7397c068cce134652d740763b"
server_key_version: "encryptionkey@0"
I20260809 20:29:38.110303 3851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:38.123720 3851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:38.127739 3851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:38.128028 3851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:38.147150 3851 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.61:37059
I20260809 20:29:38.147173 3918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.61:37059 every 8 connection(s)
I20260809 20:29:38.148633 3851 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/data/info.pb
I20260809 20:29:38.152348 3919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:38.156720 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 3851
I20260809 20:29:38.156927 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-1/wal/instance
I20260809 20:29:38.157407 12600 external_mini_cluster.cc:1428] Setting key 98d64779a2509e50940aa89c8610c071
I20260809 20:29:38.159003 3919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:29:38 dist-test-slave-rnrw krb5kdc[3759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307378, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.61@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:29:38.180339 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.161779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.12.78.62:40065)
0809 20:29:38.161966 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.161970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.163296 (+ 1326us) server_negotiation.cc:407] Connection header received
0809 20:29:38.165284 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.165290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.165382 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.165493 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.166964 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.167700 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.169310 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.169736 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.169882 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.174062 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.174067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.174071 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.174106 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.176928 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.177922 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.177926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.177928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.178048 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.178557 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.178560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.178562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.178839 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.178950 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.179937 (+ 987us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.180112 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 20:29:38.182004 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.162307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40065 (local address 127.0.0.1:51114)
0809 20:29:38.163124 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.163174 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:38.165003 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:38.165726 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:38.165733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:38.166249 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:38.166794 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.166810 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.167871 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.167875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.169159 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.169171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.169971 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.169978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.170223 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.170917 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:38.170939 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:38.173811 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:38.177089 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.177102 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.177123 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.177798 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.178156 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.178164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.178171 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.178444 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.178986 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:38.178997 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:38.179477 (+ 480us) client_negotiation.cc:769] Sending connection context
0809 20:29:38.179846 (+ 369us) client_negotiation.cc:240] Negotiation successful
0809 20:29:38.180439 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":392,"threads_started":1}
I20260809 20:29:38.183858 3919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.78.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.60@KRBTEST.COM" created.
Aug 09 20:29:38 dist-test-slave-rnrw krb5kdc[3759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307378, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.61@KRBTEST.COM for kudu/127.12.78.61@KRBTEST.COM
I20260809 20:29:38.200094 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.184875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.12.78.61:37059)
0809 20:29:38.185499 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.185504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.185530 (+ 26us) server_negotiation.cc:407] Connection header received
0809 20:29:38.185572 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.185578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.185749 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.185863 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.186884 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.187931 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.189559 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.190197 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.190354 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.193991 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.194005 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.194018 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.194051 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.197094 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.197946 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.197953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.197956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.198077 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.198498 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.198501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.198503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.198842 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.199017 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.199589 (+ 572us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.199793 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":314,"threads_started":1}
I20260809 20:29:38.200199 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.184843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:37059 (local address 127.0.0.1:56408)
0809 20:29:38.185079 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.185101 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:38.185218 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:38.185921 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:38.185928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:38.186507 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:38.186705 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.186717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.188084 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.188090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.189435 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.189447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.190408 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.190417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.190563 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.191177 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:38.191194 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:38.193750 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:38.197225 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.197232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.197238 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.197796 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.198186 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.198193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.198199 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.198383 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.198998 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:38.199005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:38.199169 (+ 164us) client_negotiation.cc:769] Sending connection context
0809 20:29:38.199322 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 20:29:38.199658 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 20:29:38.202934 3919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } has no permanent_uuid. Determining permanent_uuid...
I20260809 20:29:38.204123 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.203547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:54934)
0809 20:29:38.203767 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.203925 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260809 20:29:38.204404 3870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
I20260809 20:29:38.206030 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.205700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:54942)
0809 20:29:38.205848 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.205930 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 20:29:38.206352 3919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:41465: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.60.keytab.
Entry for principal kudu/127.12.78.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.60.keytab.
I20260809 20:29:38.228160 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.60:41465
--webserver_interface=127.12.78.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.60.keytab
--principal=kudu/127.12.78.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:40065,127.12.78.61:37059,127.12.78.60:41465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:29:38.255716 3919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } attempt: 1
I20260809 20:29:38.256983 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.256483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:54948)
0809 20:29:38.256721 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.256827 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260809 20:29:38.257990 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.257634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:54956)
0809 20:29:38.257785 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.257884 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 20:29:38.258221 3919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:41465: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
I20260809 20:29:38.326443 3919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } attempt: 2
I20260809 20:29:38.327642 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.327166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:54962)
0809 20:29:38.327367 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.327482 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 20:29:38.328713 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.328314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:54978)
0809 20:29:38.328463 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.328579 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 20:29:38.328915 3919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:41465: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
I20260809 20:29:38.333065 3844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } attempt: 5
Aug 09 20:29:38 dist-test-slave-rnrw krb5kdc[3759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307377, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for kudu/127.12.78.61@KRBTEST.COM
I20260809 20:29:38.346690 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.333716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:37059 (local address 127.0.0.1:56422)
0809 20:29:38.333897 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.333920 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:38.334045 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:38.334435 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:38.334453 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:38.334864 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:38.335063 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.335072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.336107 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.336113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.337074 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.337087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.337804 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.337810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.338022 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.338702 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:38.338721 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:38.341060 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:38.344091 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.344097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.344101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.344498 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.344966 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.344971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.344975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.345125 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.345679 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:38.345684 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:38.345856 (+ 172us) client_negotiation.cc:769] Sending connection context
0809 20:29:38.345999 (+ 143us) client_negotiation.cc:240] Negotiation successful
0809 20:29:38.346314 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 20:29:38.346679 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.333817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.12.78.61:37059)
0809 20:29:38.333973 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.333978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.334002 (+ 24us) server_negotiation.cc:407] Connection header received
0809 20:29:38.334186 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.334192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.334281 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.334390 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.335214 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.335959 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.337226 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.337649 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.337787 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.341279 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.341284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.341287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.341320 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.343944 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.344674 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.344681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.344684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.344842 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.345252 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.345257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.345259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.345549 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.345716 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.346128 (+ 412us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.346361 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 20:29:38.348011 3844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } has no permanent_uuid. Determining permanent_uuid...
I20260809 20:29:38.349079 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.348682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:54992)
0809 20:29:38.348844 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.348927 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 20:29:38.349956 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.349638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:55004)
0809 20:29:38.349772 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.349858 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 20:29:38.350234 3844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:41465: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
I20260809 20:29:38.401482 3844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } attempt: 1
I20260809 20:29:38.402740 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.402259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:51510)
0809 20:29:38.402460 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.402551 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260809 20:29:38.403741 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.403381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:51526)
0809 20:29:38.403545 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.403638 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 20:29:38.403965 3844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:41465: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
I20260809 20:29:38.420141 3919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } attempt: 3
I20260809 20:29:38.421530 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.420878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:51532)
0809 20:29:38.421156 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.421297 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260809 20:29:38.422770 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.422281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:51544)
0809 20:29:38.422513 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.422613 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260809 20:29:38.423036 3919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:41465: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
I20260809 20:29:38.472198 3844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } attempt: 2
I20260809 20:29:38.473464 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.472971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:51552)
0809 20:29:38.473199 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.473298 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 20:29:38.474495 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.474144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:51560)
0809 20:29:38.474301 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.474380 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 20:29:38.474731 3844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:41465: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
I20260809 20:29:38.566334 3919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } attempt: 4
I20260809 20:29:38.567667 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.567080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:51568)
0809 20:29:38.567362 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.567504 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 20:29:38.568813 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.568443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:51574)
0809 20:29:38.568639 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.568713 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 20:29:38.569046 3919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:41465: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
I20260809 20:29:38.580940 3844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } attempt: 3
W20260809 20:29:38.581652 3927 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 20:29:38.581997 3927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:38.582103 3927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260809 20:29:38.582080 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.581542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:51582)
0809 20:29:38.581764 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.581855 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 20:29:38.582235 3927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 20:29:38.583289 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.582847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:51592)
0809 20:29:38.583059 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.583157 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260809 20:29:38.583503 3844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:41465: Network error: Client connection negotiation failed: client connection to 127.12.78.60:41465: connect: Connection refused (error 111)
W20260809 20:29:38.590262 3927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:38.590389 3927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:38.590508 3927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:38.590595 3927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:38.590698 3927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:38.601006 3927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:40065,127.12.78.61:37059,127.12.78.60:41465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.60.keytab
--principal=kudu/127.12.78.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.60:41465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.78.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:38.602368 3927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:38.603709 3927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:38.616073 3936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:38.616036 3933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:38.616106 3934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:38.616905 3927 server_base.cc:1034] running on GCE node
I20260809 20:29:38.617830 3927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:38.619213 3927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:38.620434 3927 hybrid_clock.cc:648] HybridClock initialized: now 1786307378620378 us; error 28 us; skew 500 ppm
I20260809 20:29:38.620816 3927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:38 dist-test-slave-rnrw krb5kdc[3759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307378, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:38.624049 3927 init.cc:377] Logged in from keytab as kudu/127.12.78.60@KRBTEST.COM (short username kudu)
I20260809 20:29:38.625510 3927 webserver.cc:460] Webserver started at http://127.12.78.60:39293/ using document root <none> and password file <none>
I20260809 20:29:38.626161 3927 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:38.626266 3927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:38.626545 3927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:38.629771 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/data/instance:
uuid: "8ad80139d553473a9605d3f69b4a3978"
format_stamp: "Formatted at 2026-08-09 20:29:38 on dist-test-slave-rnrw"
server_key: "9830a21844dbf50ff26421795ded621f"
server_key_iv: "51f205037d6f5b38b6afd0c5c17b38d8"
server_key_version: "encryptionkey@0"
I20260809 20:29:38.630580 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/wal/instance:
uuid: "8ad80139d553473a9605d3f69b4a3978"
format_stamp: "Formatted at 2026-08-09 20:29:38 on dist-test-slave-rnrw"
server_key: "9830a21844dbf50ff26421795ded621f"
server_key_iv: "51f205037d6f5b38b6afd0c5c17b38d8"
server_key_version: "encryptionkey@0"
I20260809 20:29:38.636056 3927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 20:29:38.639684 3943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:38.641119 3927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 20:29:38.641325 3927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/wal
uuid: "8ad80139d553473a9605d3f69b4a3978"
format_stamp: "Formatted at 2026-08-09 20:29:38 on dist-test-slave-rnrw"
server_key: "9830a21844dbf50ff26421795ded621f"
server_key_iv: "51f205037d6f5b38b6afd0c5c17b38d8"
server_key_version: "encryptionkey@0"
I20260809 20:29:38.641566 3927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:38.655129 3927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:38.659776 3927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:38.660071 3927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:38.680091 3927 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.60:41465
I20260809 20:29:38.680158 3996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.60:41465 every 8 connection(s)
I20260809 20:29:38.681856 3927 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/data/info.pb
I20260809 20:29:38.685717 3997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:38.689256 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 3927
I20260809 20:29:38.689450 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786307001768029-12600-0/minicluster-data/master-2/wal/instance
I20260809 20:29:38.690011 12600 external_mini_cluster.cc:1428] Setting key b21a88326ef1df25d84e0b5377c74835
I20260809 20:29:38.693847 3997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:29:38 dist-test-slave-rnrw krb5kdc[3759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.60@KRBTEST.COM
Aug 09 20:29:38 dist-test-slave-rnrw krb5kdc[3759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307378, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.60@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:29:38.713385 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.697139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.12.78.62:40065)
0809 20:29:38.698536 (+ 1397us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.698541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.698556 (+ 15us) server_negotiation.cc:407] Connection header received
0809 20:29:38.698604 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.698607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.698667 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.698791 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.699915 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.700773 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.702378 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.702798 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.702943 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.706582 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.706590 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.706597 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.706644 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.710058 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.710952 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.710957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.710960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.711116 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.711539 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.711545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.711548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.711912 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.712091 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.712670 (+ 579us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.712882 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1282,"thread_start_us":1167,"threads_started":1}
I20260809 20:29:38.713948 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.697118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:40065 (local address 127.0.0.1:53694)
0809 20:29:38.697924 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.697979 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:38.698129 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:38.698853 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:38.698859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:38.699518 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:38.699752 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.699761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.700935 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.700940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.702230 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.702242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.703037 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.703044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.703208 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.703807 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:38.703857 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:38.706370 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:38.710216 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.710226 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.710231 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.710833 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.711224 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.711227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.711229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.711351 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.712089 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:38.712099 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:38.712309 (+ 210us) client_negotiation.cc:769] Sending connection context
0809 20:29:38.712711 (+ 402us) client_negotiation.cc:240] Negotiation successful
0809 20:29:38.712941 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":418,"threads_started":1}
I20260809 20:29:38.713999 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.692080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51608 (local address 127.12.78.60:41465)
0809 20:29:38.692921 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.692932 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.692976 (+ 44us) server_negotiation.cc:407] Connection header received
0809 20:29:38.694606 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.694627 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.695027 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.695404 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.697133 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.699042 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.700374 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.701105 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.701327 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.705574 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.705584 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.705600 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.705638 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.709474 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.710167 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.710177 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.710186 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.710366 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.710796 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.710800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.710802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.711555 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.711845 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.712263 (+ 418us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.712674 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":411,"threads_started":1}
I20260809 20:29:38.715596 3997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:29:38 dist-test-slave-rnrw krb5kdc[3759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307378, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.60@KRBTEST.COM for kudu/127.12.78.61@KRBTEST.COM
Aug 09 20:29:38 dist-test-slave-rnrw krb5kdc[3759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:29:38.730659 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.716440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:37059 (local address 127.0.0.1:58016)
0809 20:29:38.716714 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.716736 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:38.716848 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:38.717381 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:38.717384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:38.717712 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:38.717847 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.717855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.718811 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.718819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.720271 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.720288 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.721113 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.721119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.721303 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.721870 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:38.721887 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:38.723929 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:38.728018 (+ 4089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.728025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.728029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.728406 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.728906 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.728911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.728914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.729054 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.729704 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:38.729709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:38.729879 (+ 170us) client_negotiation.cc:769] Sending connection context
0809 20:29:38.730048 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 20:29:38.730346 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260809 20:29:38.730749 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.716472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.12.78.61:37059)
0809 20:29:38.716685 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.716690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.716786 (+ 96us) server_negotiation.cc:407] Connection header received
0809 20:29:38.716993 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.717001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.717090 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.717190 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.718000 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.718672 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.720437 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.720940 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.721095 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.724123 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.724128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.724132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.724169 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.727866 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.728526 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.728531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.728535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.728771 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.729178 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.729185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.729189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.729539 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.729694 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.730114 (+ 420us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.730333 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 20:29:38.731940 3997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } has no permanent_uuid. Determining permanent_uuid...
I20260809 20:29:38.732478 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.717270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.12.78.62:40065)
0809 20:29:38.717456 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.717461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.717478 (+ 17us) server_negotiation.cc:407] Connection header received
0809 20:29:38.717662 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.717666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.717736 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.717837 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.718989 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.720045 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.721395 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.721991 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.722131 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.726223 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.726232 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.726238 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.726285 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.729477 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.730297 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.730302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.730305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.730438 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.730899 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.730902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.730904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.731278 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.731591 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.731927 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.732125 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 20:29:38 dist-test-slave-rnrw krb5kdc[3759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307378, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.60@KRBTEST.COM for kudu/127.12.78.60@KRBTEST.COM
Aug 09 20:29:38 dist-test-slave-rnrw krb5kdc[3759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.61@KRBTEST.COM
I20260809 20:29:38.745025 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.732687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:51614)
0809 20:29:38.732860 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.732879 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:38.732990 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:38.733396 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:38.733402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:38.733756 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:38.733882 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.733889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.735091 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.735099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.735945 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.735955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.736693 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.736698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.736926 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.737498 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:38.737518 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:38.739455 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:38.742568 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.742575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.742579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.742929 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.743276 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.743282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.743285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.743419 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.743912 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:38.743917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:38.744057 (+ 140us) client_negotiation.cc:769] Sending connection context
0809 20:29:38.744204 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 20:29:38.744534 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":7}
I20260809 20:29:38.745023 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.732897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.12.78.60:41465)
0809 20:29:38.733154 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.733158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.733173 (+ 15us) server_negotiation.cc:407] Connection header received
0809 20:29:38.733210 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.733214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.733263 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.733337 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.734013 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.734977 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.736087 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.736513 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.736694 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.739636 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.739641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.739644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.739671 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.742443 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.743042 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.743049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.743052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.743172 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.743519 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.743527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.743530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.743803 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.743950 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.744279 (+ 329us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.744730 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":197,"server-negotiator.queue_time_us":147}
I20260809 20:29:38.748924 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.734194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58018 (local address 127.12.78.61:37059)
0809 20:29:38.734356 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.734361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.734380 (+ 19us) server_negotiation.cc:407] Connection header received
0809 20:29:38.734497 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.734502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.734565 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.734658 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.735795 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.736492 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.737972 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.738393 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.738530 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.743053 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.743066 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.743069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.743100 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.746254 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.746910 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.746917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.746921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.747068 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.747465 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.747471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.747474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.747746 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.748023 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.748329 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.748600 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 20:29:38.750032 3997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978: Bootstrap starting.
I20260809 20:29:38.753732 3997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:38.754217 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:38.754916 3997 log.cc:824] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:38.757608 3997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978: No bootstrap required, opened a new log
I20260809 20:29:38.756909 3817 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "97cc7bfd127146739fb4b1f167c0cec1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53698
I20260809 20:29:38.760696 3844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } attempt: 4
I20260809 20:29:38.766566 3997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } }
I20260809 20:29:38.766975 3997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad80139d553473a9605d3f69b4a3978, State: Initialized, Role: FOLLOWER
I20260809 20:29:38.767540 3997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } }
I20260809 20:29:38.768671 4004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } } }
I20260809 20:29:38.769083 4004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:38.769869 3997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978 [sys.catalog]: configured and running, proceeding with master startup.
Aug 09 20:29:38 dist-test-slave-rnrw krb5kdc[3759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307377, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for kudu/127.12.78.60@KRBTEST.COM
I20260809 20:29:38.777720 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.761293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:51622)
0809 20:29:38.761497 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.761521 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:38.761638 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:38.761962 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:38.761966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:38.762482 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:38.762703 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.762715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.764237 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.764243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.765789 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.765805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.766856 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.766863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.767153 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.767855 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:38.767878 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:38.770912 (+ 3034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:38.774832 (+ 3920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.774837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.774842 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.775235 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.775644 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.775649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.775653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.775797 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.776441 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:38.776446 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:38.776702 (+ 256us) client_negotiation.cc:769] Sending connection context
0809 20:29:38.776870 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 20:29:38.777237 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 20:29:38.778582 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.761355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51622 (local address 127.12.78.60:41465)
0809 20:29:38.761549 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.761553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.761573 (+ 20us) server_negotiation.cc:407] Connection header received
0809 20:29:38.761766 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.761770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.761834 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.761941 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.762855 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.764058 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.765943 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.766693 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.766817 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.771120 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.771126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.771129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.771164 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.774678 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.775357 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.775364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.775368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.775518 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.775912 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.775918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.775922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.776317 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.776500 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.778169 (+ 1669us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.778335 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 20:29:38.783675 3844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1: Bootstrap starting.
W20260809 20:29:38.788364 4015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 20:29:38.788523 4015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 20:29:38.787206 3927 master_runner.cc:186] Error getting master registration for 127.12.78.62:40065: OK, instance_id { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" instance_seqno: 1786307377621905 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 20:29:38.798818 3927 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 20:29:38.799510 3844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:38.800686 3844 log.cc:824] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:38.803501 3844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1: No bootstrap required, opened a new log
I20260809 20:29:38.811318 3844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } }
I20260809 20:29:38.811775 3844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cc7bfd127146739fb4b1f167c0cec1, State: Initialized, Role: FOLLOWER
I20260809 20:29:38.812258 3844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } }
I20260809 20:29:38.813354 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } } }
I20260809 20:29:38.813822 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:38.814711 3844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:38.814776 3817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 20:29:38.814973 3817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:38.819268 3817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } }
I20260809 20:29:38.821576 3817 leader_election.cc:288] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [CANDIDATE]: Term 1 election: Requested vote from peers 7a50a6848d894e6f81aa8dcef0a6c53f (127.12.78.61:37059), 8ad80139d553473a9605d3f69b4a3978 (127.12.78.60:41465)
I20260809 20:29:38.822862 3971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97cc7bfd127146739fb4b1f167c0cec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8ad80139d553473a9605d3f69b4a3978"
I20260809 20:29:38.823359 3971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:38.827070 3971 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cc7bfd127146739fb4b1f167c0cec1 in term 1.
I20260809 20:29:38.828945 3793 leader_election.cc:302] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad80139d553473a9605d3f69b4a3978, 97cc7bfd127146739fb4b1f167c0cec1; no voters:
I20260809 20:29:38.829401 4018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:38.849431 4018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [term 1 LEADER]: Becoming Leader. State: Replica: 97cc7bfd127146739fb4b1f167c0cec1, State: Running, Role: LEADER
I20260809 20:29:38.850193 4018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } }
I20260809 20:29:38.832034 3772 master_runner.cc:186] Error getting master registration for 127.12.78.61:37059: OK, instance_id { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" instance_seqno: 1786307378141825 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 20:29:38.855561 3772 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 20:29:38.857323 4030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 20:29:38.857450 4030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 20:29:38.860039 4033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97cc7bfd127146739fb4b1f167c0cec1. Latest consensus state: current_term: 1 leader_uuid: "97cc7bfd127146739fb4b1f167c0cec1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } } }
I20260809 20:29:38.860567 4033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [sys.catalog]: This master's current role is: LEADER
W20260809 20:29:38.861214 3795 tablet.cc:2369] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:29:38.868284 3919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } attempt: 5
I20260809 20:29:38.871748 4038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:38.874115 4038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 09 20:29:38 dist-test-slave-rnrw krb5kdc[3759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307378, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.61@KRBTEST.COM for kudu/127.12.78.60@KRBTEST.COM
I20260809 20:29:38.883947 3971 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978 [term 1 FOLLOWER]: Refusing update from remote peer 97cc7bfd127146739fb4b1f167c0cec1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:29:38.885032 4033 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:29:38.889308 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.868908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:41465 (local address 127.0.0.1:51632)
0809 20:29:38.869314 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:38.869336 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:38.869461 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:38.870563 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:38.870571 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:38.871301 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:38.871597 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.871625 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.873506 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.873515 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.875123 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:38.875143 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.876994 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.877004 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:38.877323 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.878135 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:38.878167 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:38.881231 (+ 3064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:38.885720 (+ 4489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.885724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.885728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.886338 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.886795 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:38.886800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:38.886804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:38.887351 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:38.888229 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:38.888234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:38.888388 (+ 154us) client_negotiation.cc:769] Sending connection context
0809 20:29:38.888535 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 20:29:38.888884 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274}
I20260809 20:29:38.890709 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.869713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.12.78.60:41465)
0809 20:29:38.869904 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.869912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.869938 (+ 26us) server_negotiation.cc:407] Connection header received
0809 20:29:38.870017 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.870023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.870124 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.870256 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.871864 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.873234 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.875639 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.876573 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.876737 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.881464 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.881474 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.881480 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.881522 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.885527 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.886487 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.886493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.886497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.886668 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.887511 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.887517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.887520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.887893 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.888150 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.889861 (+ 1711us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.890169 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 20:29:38.898844 3919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f: Bootstrap starting.
I20260809 20:29:38.904027 3919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:38.910321 3919 log.cc:824] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f: Log is configured to *not* fsync() on all Append() calls
W20260809 20:29:38.912215 3949 tablet.cc:2369] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:29:38.923719 3919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f: No bootstrap required, opened a new log
I20260809 20:29:38.933266 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.889218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.12.78.60:41465)
0809 20:29:38.904423 (+ 15205us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.904429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.906250 (+ 1821us) server_negotiation.cc:407] Connection header received
0809 20:29:38.907372 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.907378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.907488 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.907567 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.909996 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.911071 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.914239 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.915159 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.915252 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.920102 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.920108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.920124 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.920167 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.923972 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.928232 (+ 4260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.928241 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.928246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.928446 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.929455 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.929460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.929463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.929871 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.930152 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.932501 (+ 2349us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.932824 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15044,"thread_start_us":24222,"threads_started":1}
I20260809 20:29:38.938866 4004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97cc7bfd127146739fb4b1f167c0cec1. Latest consensus state: current_term: 1 leader_uuid: "97cc7bfd127146739fb4b1f167c0cec1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } } }
I20260809 20:29:38.939211 4004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:38.942991 4042 mvcc.cc:205] Tried to move back new op lower bound from 7316715023895453696 to 7316715023777239040. Current Snapshot: MvccSnapshot[applied={T|T < 7316715023895453696}]
I20260809 20:29:38.947353 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.864770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.12.78.61:37059)
0809 20:29:38.864986 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.864996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.878200 (+ 13204us) server_negotiation.cc:407] Connection header received
0809 20:29:38.878700 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.878705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.878786 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.878864 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.885720 (+ 6856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.886873 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.914992 (+ 28119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.915652 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.915764 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.934551 (+ 18787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.934561 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.934569 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.934616 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.938477 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.939548 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.939553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.939557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.939729 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.944661 (+ 4932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.944667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.944671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.945070 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.945336 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.946700 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.946958 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 20:29:38.951651 3919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } }
I20260809 20:29:38.952179 3919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a50a6848d894e6f81aa8dcef0a6c53f, State: Initialized, Role: FOLLOWER
I20260809 20:29:38.952632 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:38.874523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53704 (local address 127.12.78.62:40065)
0809 20:29:38.875479 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:38.875489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:38.881449 (+ 5960us) server_negotiation.cc:407] Connection header received
0809 20:29:38.883208 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:38.883215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:38.883306 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:38.883389 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:38.902108 (+ 18719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.903235 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.917160 (+ 13925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:38.917859 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:38.917993 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:38.940350 (+ 22357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:38.940359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:38.940363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:38.940411 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:38.944316 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.949468 (+ 5152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.949475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.949479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.949654 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:38.950230 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:38.950242 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:38.950245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:38.950653 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:38.950938 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:38.951924 (+ 986us) server_negotiation.cc:299] Negotiation successful
0809 20:29:38.952199 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755}
I20260809 20:29:38.953747 4004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97cc7bfd127146739fb4b1f167c0cec1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } } }
I20260809 20:29:38.954180 4004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:38.953078 3919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } }
I20260809 20:29:38.959420 4033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97cc7bfd127146739fb4b1f167c0cec1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } } }
I20260809 20:29:38.959714 4033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:38.959821 4038 catalog_manager.cc:1347] Generated new cluster ID: 9f400f0fe628413faea0f9eb42bb5939
I20260809 20:29:38.959926 4038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:38.960186 3919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:38.957619 4051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } } }
I20260809 20:29:38.960410 4051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:38.962014 3893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97cc7bfd127146739fb4b1f167c0cec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f"
I20260809 20:29:38.962448 3893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:38.965694 3893 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cc7bfd127146739fb4b1f167c0cec1 in term 1.
I20260809 20:29:38.966537 3892 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f [term 1 FOLLOWER]: Refusing update from remote peer 97cc7bfd127146739fb4b1f167c0cec1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:29:38.973567 4033 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:29:39.008420 4038 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:39.010481 4038 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:39.030645 4057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97cc7bfd127146739fb4b1f167c0cec1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } } }
I20260809 20:29:39.031281 4057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:39.030658 4051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97cc7bfd127146739fb4b1f167c0cec1. Latest consensus state: current_term: 1 leader_uuid: "97cc7bfd127146739fb4b1f167c0cec1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } } }
I20260809 20:29:39.031958 4051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:39.031798 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97cc7bfd127146739fb4b1f167c0cec1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cc7bfd127146739fb4b1f167c0cec1" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 40065 } } peers { permanent_uuid: "7a50a6848d894e6f81aa8dcef0a6c53f" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 37059 } } peers { permanent_uuid: "8ad80139d553473a9605d3f69b4a3978" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 41465 } } }
I20260809 20:29:39.032136 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1 [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:39.049208 4038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97cc7bfd127146739fb4b1f167c0cec1: Generated new TSK 0
I20260809 20:29:39.050375 4038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 20:29:39.072932 4067 catalog_manager.cc:1259] Loaded cluster ID: 9f400f0fe628413faea0f9eb42bb5939
I20260809 20:29:39.073128 4067 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f: loading cluster ID for follower catalog manager: success
I20260809 20:29:39.077114 4067 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f: acquiring CA information for follower catalog manager: success
I20260809 20:29:39.079852 4067 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a50a6848d894e6f81aa8dcef0a6c53f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 20:29:39.211285 3923 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:39.196967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.12.78.61:37059)
0809 20:29:39.197206 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.197213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.209057 (+ 11844us) server_negotiation.cc:407] Connection header received
0809 20:29:39.209578 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.209586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.209700 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.209780 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:39.211029 (+ 1249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58036: BlockingRecv error: recv got EOF from 127.0.0.1:58036 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260809 20:29:39.214707 4002 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:39.205213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.12.78.62:40065)
0809 20:29:39.205415 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.205421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.212423 (+ 7002us) server_negotiation.cc:407] Connection header received
0809 20:29:39.212712 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.212718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.212804 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.212879 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:39.214457 (+ 1578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53718: BlockingRecv error: recv got EOF from 127.0.0.1:53718 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260809 20:29:39.218264 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:39.215807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.12.78.60:41465)
0809 20:29:39.215982 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.215992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.216702 (+ 710us) server_negotiation.cc:407] Connection header received
0809 20:29:39.216934 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.216940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.217024 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.217094 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:39.217993 (+ 899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51654: BlockingRecv error: recv got EOF from 127.0.0.1:51654 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 20:29:39.272081 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:39.252460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.12.78.61:37059)
0809 20:29:39.252682 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.252689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.263568 (+ 10879us) server_negotiation.cc:407] Connection header received
0809 20:29:39.264183 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.264191 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.264403 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.264586 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:39.265370 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.266572 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.268798 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.269503 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.269607 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:39.270422 (+ 815us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:39.271159 (+ 737us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:39.271296 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:39.271572 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 20:29:39.271669 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 20:29:39.315613 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:39.306093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.12.78.60:41465)
0809 20:29:39.306306 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.306313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.313236 (+ 6923us) server_negotiation.cc:407] Connection header received
0809 20:29:39.313802 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.313809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.313946 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.314033 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:39.315333 (+ 1300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51668: BlockingRecv error: recv got EOF from 127.0.0.1:51668 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 20:29:39.368496 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:39.349885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.12.78.61:37059)
0809 20:29:39.350083 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.350091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.361222 (+ 11131us) server_negotiation.cc:407] Connection header received
0809 20:29:39.361782 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.361790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.361906 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.361980 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:39.362755 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.363969 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.366073 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.366802 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.366911 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:39.367596 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:39.367763 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:39.367828 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:39.368068 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 20:29:39.368160 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260809 20:29:39.411635 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:39.402384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.12.78.60:41465)
0809 20:29:39.402626 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.402636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.409535 (+ 6899us) server_negotiation.cc:407] Connection header received
0809 20:29:39.410055 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.410061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.410183 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.410262 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:39.411366 (+ 1104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51672: BlockingRecv error: recv got EOF from 127.0.0.1:51672 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 20:29:39.466516 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:39.448534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.12.78.61:37059)
0809 20:29:39.448737 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.448744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.459648 (+ 10904us) server_negotiation.cc:407] Connection header received
0809 20:29:39.460164 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.460170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.460271 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.460334 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:39.461062 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.462200 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.464150 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.464854 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.464959 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:39.465584 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:39.465739 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:39.465805 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:39.466078 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 20:29:39.466173 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 20:29:39.510291 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:39.501079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.12.78.60:41465)
0809 20:29:39.501274 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.501282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.508275 (+ 6993us) server_negotiation.cc:407] Connection header received
0809 20:29:39.508535 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.508560 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.508661 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.508746 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:39.510014 (+ 1268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51680: BlockingRecv error: recv got EOF from 127.0.0.1:51680 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 20:29:39.528373 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:39.518395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58068 (local address 127.12.78.61:37059)
0809 20:29:39.518729 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.518736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.521503 (+ 2767us) server_negotiation.cc:407] Connection header received
0809 20:29:39.521742 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.521749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.521849 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.521967 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:39.522487 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.523658 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.526179 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.526930 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.527081 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:39.527552 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:39.527703 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:39.527801 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:39.527953 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 20:29:39.528059 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
W20260809 20:29:39.538120 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:39.535445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.12.78.60:41465)
0809 20:29:39.535958 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.535965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.536528 (+ 563us) server_negotiation.cc:407] Connection header received
0809 20:29:39.536729 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.536736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.536845 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.536957 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:39.537866 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51684: BlockingRecv error: recv got EOF from 127.0.0.1:51684 (error 108)
Metrics: {"server-negotiator.queue_time_us":358}
I20260809 20:29:39.555814 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:39.549514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.12.78.61:37059)
0809 20:29:39.549736 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.549743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.550024 (+ 281us) server_negotiation.cc:407] Connection header received
0809 20:29:39.550183 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.550189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.550283 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.550409 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:39.550753 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.551944 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.553694 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.554362 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.554510 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:39.554956 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:39.555101 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:39.555202 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:39.555332 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 20:29:39.555427 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260809 20:29:39.564427 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:39.562611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.12.78.60:41465)
0809 20:29:39.562804 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.562811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.563100 (+ 289us) server_negotiation.cc:407] Connection header received
0809 20:29:39.563246 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.563252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.563350 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.563485 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:39.564173 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51700: BlockingRecv error: recv got EOF from 127.0.0.1:51700 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 20:29:39.589918 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:39.584432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.12.78.61:37059)
0809 20:29:39.584689 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.584696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.585076 (+ 380us) server_negotiation.cc:407] Connection header received
0809 20:29:39.585307 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.585312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.585382 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.585487 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:39.585887 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.586678 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.588022 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.588487 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.588706 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:39.589140 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:39.589289 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:39.589393 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:39.589540 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 20:29:39.589621 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260809 20:29:39.597817 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:39.595815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.12.78.60:41465)
0809 20:29:39.596040 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.596046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.596460 (+ 414us) server_negotiation.cc:407] Connection header received
0809 20:29:39.596604 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.596610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.596730 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.596845 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:39.597577 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51708: BlockingRecv error: recv got EOF from 127.0.0.1:51708 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 20:29:39.642915 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:39.635907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.12.78.61:37059)
0809 20:29:39.636107 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.636113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.636681 (+ 568us) server_negotiation.cc:407] Connection header received
0809 20:29:39.636885 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.636896 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.637015 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.637122 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:39.637594 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.638757 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.640663 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.641402 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.641560 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:39.642016 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:39.642175 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:39.642278 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:39.642452 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 20:29:39.642559 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260809 20:29:39.653507 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:39.651449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.12.78.60:41465)
0809 20:29:39.651696 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.651702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.651963 (+ 261us) server_negotiation.cc:407] Connection header received
0809 20:29:39.652181 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.652185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.652257 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.652365 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:39.653283 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51714: BlockingRecv error: recv got EOF from 127.0.0.1:51714 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260809 20:29:39.726954 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:39.721424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.12.78.61:37059)
0809 20:29:39.721658 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.721667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.721982 (+ 315us) server_negotiation.cc:407] Connection header received
0809 20:29:39.722226 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.722231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.722298 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.722397 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:39.722818 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.723719 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.725239 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.725673 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.725808 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:39.726234 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:39.726340 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:39.726443 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:39.726608 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 20:29:39.726689 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260809 20:29:39.734403 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:39.732509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.12.78.60:41465)
0809 20:29:39.732759 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.732765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.733032 (+ 267us) server_negotiation.cc:407] Connection header received
0809 20:29:39.733182 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.733186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.733270 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.733353 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:39.734142 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51722: BlockingRecv error: recv got EOF from 127.0.0.1:51722 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 20:29:39.790519 4015 catalog_manager.cc:1259] Loaded cluster ID: 9f400f0fe628413faea0f9eb42bb5939
I20260809 20:29:39.790701 4015 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978: loading cluster ID for follower catalog manager: success
I20260809 20:29:39.793286 4015 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978: acquiring CA information for follower catalog manager: success
I20260809 20:29:39.794974 4015 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8ad80139d553473a9605d3f69b4a3978: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 20:29:39.871290 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:39.866044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.12.78.61:37059)
0809 20:29:39.866309 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.866318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.866569 (+ 251us) server_negotiation.cc:407] Connection header received
0809 20:29:39.866821 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.866825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.866893 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.866994 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:39.867402 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.868121 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.869495 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.869955 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.870088 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:39.870522 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:39.870667 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:39.870774 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:39.870929 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 20:29:39.871011 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 20:29:39.882217 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:39.876923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51738 (local address 127.12.78.60:41465)
0809 20:29:39.877091 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.877096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.877512 (+ 416us) server_negotiation.cc:407] Connection header received
0809 20:29:39.877650 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.877655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.877813 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.877909 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:39.878259 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.878945 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.880169 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.880631 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.880726 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:39.881202 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:39.881724 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:39.881833 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:39.881925 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 20:29:39.881989 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 20:29:39.896881 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:39.888703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53728 (local address 127.12.78.62:40065)
0809 20:29:39.889590 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.889596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.890384 (+ 788us) server_negotiation.cc:407] Connection header received
0809 20:29:39.890693 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.890702 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.890947 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.891158 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:39.891676 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.892615 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.894343 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.894772 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.894918 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:39.895507 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:39.895967 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:39.896080 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:39.896383 (+ 303us) server_negotiation.cc:299] Negotiation successful
0809 20:29:39.896475 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":592,"threads_started":1}
I20260809 20:29:39.897362 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:39.888906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.12.78.61:37059)
0809 20:29:39.889141 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.889149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.890757 (+ 1608us) server_negotiation.cc:407] Connection header received
0809 20:29:39.891079 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.891087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.891187 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.891254 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:39.892277 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.893307 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.895484 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.895939 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.896080 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:39.896497 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:39.896674 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:39.896762 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:39.896989 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 20:29:39.897072 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 20:29:39.897925 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:39.890149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.12.78.60:41465)
0809 20:29:39.890322 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:39.890330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:39.890738 (+ 408us) server_negotiation.cc:407] Connection header received
0809 20:29:39.890934 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:39.890942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:39.891037 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:39.891130 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:39.892241 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.893122 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.895438 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:39.896051 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:39.896165 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:39.897216 (+ 1051us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:39.897348 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:39.897417 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:39.897665 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 20:29:39.897741 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 20:29:39.902313 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 3772
I20260809 20:29:39.903395 3838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:29:40.014559 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 3772
W20260809 20:29:40.037551 3870 connection.cc:535] server connection from 127.0.0.1:56422 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 20:29:40.037997 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 3851
I20260809 20:29:40.039081 3914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:29:40.145895 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 3851
I20260809 20:29:40.168604 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 3927
I20260809 20:29:40.169754 3992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:29:40.277464 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 3927
2026-08-09T20:29:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4587 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 20:29:40 dist-test-slave-rnrw krb5kdc[4187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 20:29:40 dist-test-slave-rnrw krb5kdc[4187](info): set up 2 sockets
Aug 09 20:29:40 dist-test-slave-rnrw krb5kdc[4187](info): commencing operation
krb5kdc: starting...
W20260809 20:29:41.428378 12600 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.005s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/test-user.keytab.
Aug 09 20:29:41 dist-test-slave-rnrw krb5kdc[4187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T20:29:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T20:29:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.78.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
Entry for principal kudu/127.12.78.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab.
I20260809 20:29:41.742235 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:43109
--webserver_interface=127.12.78.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:43109,127.12.78.61:44675,127.12.78.60:38075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 20:29:42.095837 4200 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 20:29:42.096175 4200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:42.096305 4200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:42.096439 4200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:42.103780 4200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:42.103860 4200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:42.103917 4200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:42.103986 4200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:42.104079 4200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:42.113896 4200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:43109,127.12.78.61:44675,127.12.78.60:38075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.62.keytab
--principal=kudu/127.12.78.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.62:43109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.78.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:42.115173 4200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:42.116456 4200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:42.130550 4205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:42.130484 4209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:42.130508 4207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:42.132012 4200 server_base.cc:1034] running on GCE node
I20260809 20:29:42.132895 4200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:42.134332 4200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:42.135584 4200 hybrid_clock.cc:648] HybridClock initialized: now 1786307382135514 us; error 42 us; skew 500 ppm
I20260809 20:29:42.135933 4200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:42 dist-test-slave-rnrw krb5kdc[4187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307382, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:42.139272 4200 init.cc:377] Logged in from keytab as kudu/127.12.78.62@KRBTEST.COM (short username kudu)
I20260809 20:29:42.140848 4200 webserver.cc:460] Webserver started at http://127.12.78.62:39189/ using document root <none> and password file <none>
I20260809 20:29:42.141539 4200 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:42.141692 4200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:42.142030 4200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:42.145114 4200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/data/instance:
uuid: "83f95430bd464b878c3d342e82a81efd"
format_stamp: "Formatted at 2026-08-09 20:29:42 on dist-test-slave-rnrw"
server_key: "40f166064bb963c44ec3fb8906871f5e"
server_key_iv: "3d30baa7ab96054ddd85ce964b04e91f"
server_key_version: "encryptionkey@0"
I20260809 20:29:42.145936 4200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/wal/instance:
uuid: "83f95430bd464b878c3d342e82a81efd"
format_stamp: "Formatted at 2026-08-09 20:29:42 on dist-test-slave-rnrw"
server_key: "40f166064bb963c44ec3fb8906871f5e"
server_key_iv: "3d30baa7ab96054ddd85ce964b04e91f"
server_key_version: "encryptionkey@0"
I20260809 20:29:42.151103 4200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 20:29:42.154520 4217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:42.155798 4200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 20:29:42.155992 4200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/wal
uuid: "83f95430bd464b878c3d342e82a81efd"
format_stamp: "Formatted at 2026-08-09 20:29:42 on dist-test-slave-rnrw"
server_key: "40f166064bb963c44ec3fb8906871f5e"
server_key_iv: "3d30baa7ab96054ddd85ce964b04e91f"
server_key_version: "encryptionkey@0"
I20260809 20:29:42.156219 4200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:42.179433 4200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:42.184010 4200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:42.184311 4200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:42.204416 4200 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.62:43109
I20260809 20:29:42.204439 4273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.62:43109 every 8 connection(s)
I20260809 20:29:42.205883 4200 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/data/info.pb
I20260809 20:29:42.209728 4275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:42.213781 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 4200
I20260809 20:29:42.213999 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-0/wal/instance
I20260809 20:29:42.214548 12600 external_mini_cluster.cc:1428] Setting key 6adb4c2c619349ee64e9d1a32cad3574
I20260809 20:29:42.218295 4275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:29:42 dist-test-slave-rnrw krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307382, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:29:42.245983 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.222324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:43109 (local address 127.0.0.1:38660)
0809 20:29:42.223269 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.223326 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:42.226317 (+ 2991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:42.227267 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:42.227277 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:42.228143 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:42.228690 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:42.228710 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.231082 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.231087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:42.232185 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:42.232197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.233641 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.233646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:42.233793 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:42.234486 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:42.234509 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:42.236842 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:42.241563 (+ 4721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:42.241574 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:42.241584 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:42.241981 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:42.242539 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:42.242542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:42.242545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:42.242718 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:42.243803 (+ 1085us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:42.243810 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:42.244009 (+ 199us) client_negotiation.cc:769] Sending connection context
0809 20:29:42.244227 (+ 218us) client_negotiation.cc:240] Negotiation successful
0809 20:29:42.244725 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":372,"threads_started":1}
I20260809 20:29:42.246014 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.223581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.12.78.62:43109)
0809 20:29:42.225135 (+ 1554us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:42.225141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:42.225186 (+ 45us) server_negotiation.cc:407] Connection header received
0809 20:29:42.226720 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:42.226726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:42.227000 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:42.227176 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:42.228917 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.230885 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.232346 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.233393 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.233610 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:42.237100 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:42.237122 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:42.237141 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:42.237186 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:42.241416 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:42.242150 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:42.242163 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:42.242169 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:42.242394 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:42.242847 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:42.242855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:42.242861 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:42.243649 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:42.243873 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:42.244441 (+ 568us) server_negotiation.cc:299] Negotiation successful
0809 20:29:42.244945 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":1382,"thread_start_us":1205,"threads_started":1}
WARNING: no policy specified for kudu/127.12.78.61@KRBTEST.COM; defaulting to no policy
I20260809 20:29:42.249559 4275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } has no permanent_uuid. Determining permanent_uuid...
I20260809 20:29:42.250873 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.250338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:44675 (local address 127.0.0.1:44594)
0809 20:29:42.250560 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.250726 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 20:29:42.251210 4220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.61@KRBTEST.COM" created.
I20260809 20:29:42.252884 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.252450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:44675 (local address 127.0.0.1:44598)
0809 20:29:42.252639 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.252725 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 20:29:42.253165 4275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:44675: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.61.keytab.
Entry for principal kudu/127.12.78.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.61.keytab.
I20260809 20:29:42.289229 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.61:44675
--webserver_interface=127.12.78.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.61.keytab
--principal=kudu/127.12.78.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:43109,127.12.78.61:44675,127.12.78.60:38075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:29:42.302513 4275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } attempt: 1
I20260809 20:29:42.303666 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.303211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:44675 (local address 127.0.0.1:44614)
0809 20:29:42.303415 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.303512 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 20:29:42.304791 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.304373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:44675 (local address 127.0.0.1:44622)
0809 20:29:42.304529 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.304637 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 20:29:42.304961 4275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:44675: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
I20260809 20:29:42.373216 4275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } attempt: 2
I20260809 20:29:42.374411 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.373955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:44675 (local address 127.0.0.1:44628)
0809 20:29:42.374161 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.374265 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 20:29:42.375543 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.375171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:44675 (local address 127.0.0.1:44642)
0809 20:29:42.375330 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.375407 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 20:29:42.375718 4275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:44675: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
I20260809 20:29:42.467011 4275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } attempt: 3
I20260809 20:29:42.468425 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.467956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:44675 (local address 127.0.0.1:44652)
0809 20:29:42.468173 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.468286 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 20:29:42.469589 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.469174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:44675 (local address 127.0.0.1:44664)
0809 20:29:42.469346 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.469410 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 20:29:42.469789 4275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:44675: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
I20260809 20:29:42.613086 4275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } attempt: 4
I20260809 20:29:42.614307 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.613764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:44675 (local address 127.0.0.1:44680)
0809 20:29:42.614074 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.614178 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260809 20:29:42.615316 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.614969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:44675 (local address 127.0.0.1:44692)
0809 20:29:42.615122 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.615203 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 20:29:42.615542 4275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.61:44675: Network error: Client connection negotiation failed: client connection to 127.12.78.61:44675: connect: Connection refused (error 111)
W20260809 20:29:42.635185 4283 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 20:29:42.635550 4283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:42.635677 4283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:42.635816 4283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 20:29:42.642951 4283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:42.643069 4283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:42.643178 4283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:42.643275 4283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:42.643366 4283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:42.653173 4283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:43109,127.12.78.61:44675,127.12.78.60:38075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.61.keytab
--principal=kudu/127.12.78.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.61:44675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.78.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:42.654510 4283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:42.655897 4283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:42.667766 4289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:42.667797 4292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:42.667738 4290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:42.669184 4283 server_base.cc:1034] running on GCE node
I20260809 20:29:42.670017 4283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:42.671370 4283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:42.672672 4283 hybrid_clock.cc:648] HybridClock initialized: now 1786307382672585 us; error 39 us; skew 500 ppm
I20260809 20:29:42.673017 4283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:42 dist-test-slave-rnrw krb5kdc[4187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307382, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:42.683516 4283 init.cc:377] Logged in from keytab as kudu/127.12.78.61@KRBTEST.COM (short username kudu)
I20260809 20:29:42.685199 4283 webserver.cc:460] Webserver started at http://127.12.78.61:39465/ using document root <none> and password file <none>
I20260809 20:29:42.685917 4283 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:42.686048 4283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:42.686385 4283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:42.689497 4283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/data/instance:
uuid: "ca1060892656480ba8e5fe548de1a4d6"
format_stamp: "Formatted at 2026-08-09 20:29:42 on dist-test-slave-rnrw"
server_key: "41245c4a420e6ed80dd68079a76c2be9"
server_key_iv: "d830c3c990071c451d1f948e6b2f84e5"
server_key_version: "encryptionkey@0"
I20260809 20:29:42.690292 4283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/wal/instance:
uuid: "ca1060892656480ba8e5fe548de1a4d6"
format_stamp: "Formatted at 2026-08-09 20:29:42 on dist-test-slave-rnrw"
server_key: "41245c4a420e6ed80dd68079a76c2be9"
server_key_iv: "d830c3c990071c451d1f948e6b2f84e5"
server_key_version: "encryptionkey@0"
I20260809 20:29:42.695293 4283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 20:29:42.698668 4301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:42.699975 4283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 20:29:42.700160 4283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/wal
uuid: "ca1060892656480ba8e5fe548de1a4d6"
format_stamp: "Formatted at 2026-08-09 20:29:42 on dist-test-slave-rnrw"
server_key: "41245c4a420e6ed80dd68079a76c2be9"
server_key_iv: "d830c3c990071c451d1f948e6b2f84e5"
server_key_version: "encryptionkey@0"
I20260809 20:29:42.700373 4283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:42.712749 4283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:42.716482 4283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:42.716814 4283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:42.735268 4283 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.61:44675
I20260809 20:29:42.735287 4356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.61:44675 every 8 connection(s)
I20260809 20:29:42.736724 4283 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/data/info.pb
I20260809 20:29:42.740074 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 4283
I20260809 20:29:42.740280 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-1/wal/instance
I20260809 20:29:42.740386 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:42.740829 12600 external_mini_cluster.cc:1428] Setting key 6b0e7660682444f227fcaa538d4601c3
I20260809 20:29:42.746718 4357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:29:42 dist-test-slave-rnrw krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307382, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.61@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:29:42.769668 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.749185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.12.78.62:43109)
0809 20:29:42.749696 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:42.749703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:42.750644 (+ 941us) server_negotiation.cc:407] Connection header received
0809 20:29:42.753257 (+ 2613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:42.753265 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:42.753363 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:42.753493 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:42.754985 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.756200 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.757762 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.758501 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.758644 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:42.762251 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:42.762260 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:42.762268 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:42.762332 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:42.766050 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:42.766920 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:42.766928 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:42.766934 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:42.767083 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:42.767522 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:42.767528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:42.767533 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:42.767947 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:42.768122 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:42.768874 (+ 752us) server_negotiation.cc:299] Negotiation successful
0809 20:29:42.769144 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":239,"threads_started":1}
I20260809 20:29:42.770252 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.749659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:43109 (local address 127.0.0.1:38670)
0809 20:29:42.750454 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.750512 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:42.752987 (+ 2475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:42.753719 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:42.753727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:42.754234 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:42.754766 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:42.754783 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.756360 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.756364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:42.757612 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:42.757624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.758706 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.758711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:42.758893 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:42.759657 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:42.759684 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:42.762013 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:42.766222 (+ 4209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:42.766229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:42.766241 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:42.766775 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:42.767199 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:42.767202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:42.767204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:42.767383 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:42.768085 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:42.768092 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:42.768417 (+ 325us) client_negotiation.cc:769] Sending connection context
0809 20:29:42.768717 (+ 300us) client_negotiation.cc:240] Negotiation successful
0809 20:29:42.769121 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":279,"threads_started":1}
WARNING: no policy specified for kudu/127.12.78.60@KRBTEST.COM; defaulting to no policy
I20260809 20:29:42.772097 4357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.78.60@KRBTEST.COM" created.
Aug 09 20:29:42 dist-test-slave-rnrw krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307382, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.61@KRBTEST.COM for kudu/127.12.78.61@KRBTEST.COM
I20260809 20:29:42.788568 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.772921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:44675 (local address 127.0.0.1:44702)
0809 20:29:42.773102 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.773123 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:42.773226 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:42.774271 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:42.774275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:42.774641 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:42.774852 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:42.774867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.776302 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.776310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:42.777196 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:42.777207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.778459 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.778465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:42.778637 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:42.779247 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:42.779269 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:42.782040 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:42.786039 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:42.786044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:42.786051 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:42.786490 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:42.786856 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:42.786863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:42.786866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:42.786998 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:42.787582 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:42.787591 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:42.787762 (+ 171us) client_negotiation.cc:769] Sending connection context
0809 20:29:42.787901 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 20:29:42.788169 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 20:29:42.788700 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.773231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44702 (local address 127.12.78.61:44675)
0809 20:29:42.773829 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:42.773834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:42.773859 (+ 25us) server_negotiation.cc:407] Connection header received
0809 20:29:42.773898 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:42.773903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:42.774060 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:42.774222 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:42.775048 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.776143 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.777332 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.778308 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.778441 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:42.782244 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:42.782258 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:42.782281 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:42.782322 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:42.785893 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:42.786621 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:42.786630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:42.786633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:42.786752 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:42.787098 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:42.787106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:42.787109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:42.787459 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:42.787672 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:42.788064 (+ 392us) server_negotiation.cc:299] Negotiation successful
0809 20:29:42.788280 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":322,"threads_started":1}
I20260809 20:29:42.791116 4357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } has no permanent_uuid. Determining permanent_uuid...
I20260809 20:29:42.792348 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.791759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34688)
0809 20:29:42.791990 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.792137 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 20:29:42.792637 4303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
I20260809 20:29:42.794317 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.793857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34694)
0809 20:29:42.794044 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.794162 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 20:29:42.794569 4357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:38075: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.78.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.60.keytab.
Entry for principal kudu/127.12.78.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.60.keytab.
I20260809 20:29:42.813091 12600 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
/tmp/dist-test-tasktNpCrv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.60:38075
--webserver_interface=127.12.78.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.60.keytab
--principal=kudu/127.12.78.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.78.20:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.78.62:43109,127.12.78.61:44675,127.12.78.60:38075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 20:29:42.843997 4357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } attempt: 1
I20260809 20:29:42.845304 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.844756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34698)
0809 20:29:42.844984 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.845115 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 20:29:42.846400 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.845997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34708)
0809 20:29:42.846150 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.846263 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 20:29:42.846623 4357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:38075: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
I20260809 20:29:42.914781 4275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } attempt: 5
I20260809 20:29:42.914840 4357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } attempt: 2
I20260809 20:29:42.916164 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.915568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34720)
0809 20:29:42.915798 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.915922 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 20:29:42.917407 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.917008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34728)
0809 20:29:42.917179 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.917271 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 20:29:42.918666 4357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:38075: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Aug 09 20:29:42 dist-test-slave-rnrw krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307382, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for kudu/127.12.78.61@KRBTEST.COM
I20260809 20:29:42.929654 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.915570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:44675 (local address 127.0.0.1:44706)
0809 20:29:42.915858 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.915877 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:42.916001 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:42.916446 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:42.916452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:42.916895 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:42.917123 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:42.917135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.918543 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.918548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:42.919556 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:42.919568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.920514 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.920519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:42.920752 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:42.921476 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:42.921498 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:42.923536 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:42.927356 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:42.927360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:42.927363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:42.927768 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:42.928172 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:42.928175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:42.928177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:42.928308 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:42.928935 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:42.928938 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:42.929073 (+ 135us) client_negotiation.cc:769] Sending connection context
0809 20:29:42.929219 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 20:29:42.929414 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260809 20:29:42.929843 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.915742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.12.78.61:44675)
0809 20:29:42.915898 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:42.915906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:42.915929 (+ 23us) server_negotiation.cc:407] Connection header received
0809 20:29:42.916171 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:42.916175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:42.916285 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:42.916386 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:42.917281 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.918403 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.919699 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:42.920347 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:42.920497 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:42.923729 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:42.923735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:42.923740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:42.923778 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:42.927207 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:42.927889 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:42.927895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:42.927899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:42.928061 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:42.928410 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:42.928415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:42.928418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:42.928801 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:42.928948 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:42.929340 (+ 392us) server_negotiation.cc:299] Negotiation successful
0809 20:29:42.929560 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 20:29:42.930856 4275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } has no permanent_uuid. Determining permanent_uuid...
I20260809 20:29:42.931787 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.931425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34740)
0809 20:29:42.931581 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.931659 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 20:29:42.932715 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.932354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34748)
0809 20:29:42.932486 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.932618 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 20:29:42.932940 4275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:38075: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
I20260809 20:29:42.984169 4275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } attempt: 1
I20260809 20:29:42.985507 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.985027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34756)
0809 20:29:42.985236 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.985332 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260809 20:29:42.986569 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:42.986162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34758)
0809 20:29:42.986318 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:42.986432 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 20:29:42.986779 4275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:38075: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
I20260809 20:29:43.009912 4357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } attempt: 3
I20260809 20:29:43.011085 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.010593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34760)
0809 20:29:43.010819 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:43.010926 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260809 20:29:43.012353 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.011887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34770)
0809 20:29:43.012158 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:43.012239 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260809 20:29:43.012636 4357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:38075: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
I20260809 20:29:43.055037 4275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } attempt: 2
I20260809 20:29:43.056232 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.055717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34782)
0809 20:29:43.055967 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:43.056071 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 20:29:43.057312 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.056982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34788)
0809 20:29:43.057135 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:43.057209 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 20:29:43.057554 4275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:38075: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
I20260809 20:29:43.155900 4357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } attempt: 4
I20260809 20:29:43.157167 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.156671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34792)
0809 20:29:43.156923 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:43.157013 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 20:29:43.158255 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.157859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34802)
0809 20:29:43.158069 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:43.158141 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260809 20:29:43.158560 4357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:38075: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
W20260809 20:29:43.160000 4365 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 20:29:43.160378 4365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 20:29:43.160507 4365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 20:29:43.160696 4365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 20:29:43.163780 4275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } attempt: 3
I20260809 20:29:43.164798 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.164322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34810)
0809 20:29:43.164499 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:43.164642 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 20:29:43.165778 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.165417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34822)
0809 20:29:43.165609 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:43.165682 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260809 20:29:43.166010 4275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.78.60:38075: Network error: Client connection negotiation failed: client connection to 127.12.78.60:38075: connect: Connection refused (error 111)
W20260809 20:29:43.168154 4365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 20:29:43.168232 4365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 20:29:43.168332 4365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 20:29:43.168406 4365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 20:29:43.168516 4365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 20:29:43.178674 4365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.78.20:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.78.62:43109,127.12.78.61:44675,127.12.78.60:38075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/krb5kdc/kudu_127.12.78.60.keytab
--principal=kudu/127.12.78.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.78.60:38075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.78.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 18:55:23 UTC on ccycloud.quasar-joeceo.root.comops.site
build id 4576685
ASAN enabled
I20260809 20:29:43.180027 4365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 20:29:43.181445 4365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 20:29:43.193897 4372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:43.193920 4374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 20:29:43.193979 4371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 20:29:43.194566 4365 server_base.cc:1034] running on GCE node
I20260809 20:29:43.195412 4365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 20:29:43.196815 4365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 20:29:43.198060 4365 hybrid_clock.cc:648] HybridClock initialized: now 1786307383197958 us; error 56 us; skew 500 ppm
I20260809 20:29:43.198411 4365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 20:29:43 dist-test-slave-rnrw krb5kdc[4187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307383, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 20:29:43.201623 4365 init.cc:377] Logged in from keytab as kudu/127.12.78.60@KRBTEST.COM (short username kudu)
I20260809 20:29:43.203132 4365 webserver.cc:460] Webserver started at http://127.12.78.60:44795/ using document root <none> and password file <none>
I20260809 20:29:43.203969 4365 fs_manager.cc:359] Metadata directory not provided
I20260809 20:29:43.204090 4365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 20:29:43.204496 4365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 20:29:43.207464 4365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/data/instance:
uuid: "51ad88be2e314663a55e07359393706f"
format_stamp: "Formatted at 2026-08-09 20:29:43 on dist-test-slave-rnrw"
server_key: "9ebd7f62cbb54876763b29b876c5f8be"
server_key_iv: "a468c7d8e0237ac44709bd6ce0dda4a6"
server_key_version: "encryptionkey@0"
I20260809 20:29:43.208262 4365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/wal/instance:
uuid: "51ad88be2e314663a55e07359393706f"
format_stamp: "Formatted at 2026-08-09 20:29:43 on dist-test-slave-rnrw"
server_key: "9ebd7f62cbb54876763b29b876c5f8be"
server_key_iv: "a468c7d8e0237ac44709bd6ce0dda4a6"
server_key_version: "encryptionkey@0"
I20260809 20:29:43.213403 4365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 20:29:43.216763 4381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 20:29:43.218056 4365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 20:29:43.218241 4365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/data,/tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/wal
uuid: "51ad88be2e314663a55e07359393706f"
format_stamp: "Formatted at 2026-08-09 20:29:43 on dist-test-slave-rnrw"
server_key: "9ebd7f62cbb54876763b29b876c5f8be"
server_key_iv: "a468c7d8e0237ac44709bd6ce0dda4a6"
server_key_version: "encryptionkey@0"
I20260809 20:29:43.218461 4365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 20:29:43.251865 4365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 20:29:43.255888 4365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 20:29:43.256178 4365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 20:29:43.275280 4365 rpc_server.cc:307] RPC server started. Bound to: 127.12.78.60:38075
I20260809 20:29:43.275290 4439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.78.60:38075 every 8 connection(s)
I20260809 20:29:43.276805 4365 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/data/info.pb
I20260809 20:29:43.280522 4440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 20:29:43.281867 12600 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu as pid 4365
I20260809 20:29:43.282073 12600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktNpCrv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786307001768029-12600-0/minicluster-data/master-2/wal/instance
I20260809 20:29:43.282696 12600 external_mini_cluster.cc:1428] Setting key b4975548e19f625c5c1103925cefd294
I20260809 20:29:43.290151 4440 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:29:43 dist-test-slave-rnrw krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.60@KRBTEST.COM
Aug 09 20:29:43 dist-test-slave-rnrw krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307383, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.60@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
I20260809 20:29:43.306704 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.284524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.12.78.60:38075)
0809 20:29:43.285347 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.285358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.285399 (+ 41us) server_negotiation.cc:407] Connection header received
0809 20:29:43.287040 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.287061 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.287460 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.287785 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.289619 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.290926 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.292700 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.294174 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.294373 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.298529 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:43.298536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:43.298546 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:43.298584 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:43.302105 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.302808 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.302818 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.302831 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.302971 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.303366 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.303372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.303376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.304006 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:43.304343 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:43.304960 (+ 617us) server_negotiation.cc:299] Negotiation successful
0809 20:29:43.305380 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":343,"threads_started":1}
I20260809 20:29:43.309903 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.292987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.62:43109 (local address 127.0.0.1:38672)
0809 20:29:43.294135 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:43.294187 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:43.294377 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:43.294825 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:43.294832 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:43.295527 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:43.295797 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:43.295807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.297121 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.297127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:43.298325 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:43.298339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.299203 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.299208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:43.299456 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.300106 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:43.300140 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:43.302610 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:43.306370 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:43.306379 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:43.306384 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:43.306787 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:43.308079 (+ 1292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:43.308084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:43.308088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:43.308262 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:43.308819 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:43.308825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:43.309034 (+ 209us) client_negotiation.cc:769] Sending connection context
0809 20:29:43.309247 (+ 213us) client_negotiation.cc:240] Negotiation successful
0809 20:29:43.309483 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":462,"threads_started":1}
I20260809 20:29:43.310014 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.293007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.12.78.62:43109)
0809 20:29:43.293776 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.293783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.294421 (+ 638us) server_negotiation.cc:407] Connection header received
0809 20:29:43.294515 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.294521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.294620 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.294754 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.295983 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.296955 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.298561 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.298990 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.299152 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.302819 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:43.302824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:43.302828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:43.302861 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:43.306233 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.307831 (+ 1598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.307835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.307837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.307961 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.308370 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.308373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.308375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.308692 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:43.308805 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:43.309380 (+ 575us) server_negotiation.cc:299] Negotiation successful
0809 20:29:43.309585 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":494,"threads_started":1}
I20260809 20:29:43.311684 4440 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:29:43 dist-test-slave-rnrw krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.62@KRBTEST.COM
Aug 09 20:29:43 dist-test-slave-rnrw krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307383, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.60@KRBTEST.COM for kudu/127.12.78.61@KRBTEST.COM
I20260809 20:29:43.325591 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.310075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.12.78.62:43109)
0809 20:29:43.310662 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.310668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.310691 (+ 23us) server_negotiation.cc:407] Connection header received
0809 20:29:43.310735 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.310738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.310805 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.310911 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.312365 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.313354 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.315015 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.315685 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.315807 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.320139 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:43.320144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:43.320156 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:43.320185 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:43.322860 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.323543 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.323550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.323553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.323684 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.324053 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.324059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.324063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.324327 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:43.324655 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:43.324963 (+ 308us) server_negotiation.cc:299] Negotiation successful
0809 20:29:43.325215 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":264,"threads_started":1}
I20260809 20:29:43.326258 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.312460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44710 (local address 127.12.78.61:44675)
0809 20:29:43.312789 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.312794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.312813 (+ 19us) server_negotiation.cc:407] Connection header received
0809 20:29:43.312863 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.312866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.312924 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.313003 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.313811 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.314839 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.316350 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.316834 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.316966 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.320469 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:43.320474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:43.320477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:43.320510 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:43.323687 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.324304 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.324311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.324314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.324445 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.324855 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.324861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.324865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.325119 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:43.325296 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:43.325794 (+ 498us) server_negotiation.cc:299] Negotiation successful
0809 20:29:43.325964 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260809 20:29:43.326283 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.312449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.61:44675 (local address 127.0.0.1:44710)
0809 20:29:43.312657 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:43.312676 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:43.312769 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:43.313053 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:43.313057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:43.313510 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:43.313656 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:43.313664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.315011 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.315016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:43.316204 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:43.316215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.316992 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.316998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:43.317170 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.317697 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:43.317712 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:43.320280 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:43.323818 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:43.323824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:43.323828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:43.324186 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:43.324584 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:43.324589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:43.324593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:43.324741 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:43.325365 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:43.325370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:43.325519 (+ 149us) client_negotiation.cc:769] Sending connection context
0809 20:29:43.325646 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 20:29:43.325962 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 20:29:43.327448 4440 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 20:29:43 dist-test-slave-rnrw krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307383, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.60@KRBTEST.COM for kudu/127.12.78.60@KRBTEST.COM
Aug 09 20:29:43 dist-test-slave-rnrw krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.78.61@KRBTEST.COM
I20260809 20:29:43.341403 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.328209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34840 (local address 127.12.78.60:38075)
0809 20:29:43.328356 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.328360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.328374 (+ 14us) server_negotiation.cc:407] Connection header received
0809 20:29:43.328462 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.328465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.328513 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.328634 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.329506 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.330255 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.331543 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.331990 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.332145 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.335548 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:43.335552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:43.335556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:43.335583 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:43.338416 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.339356 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.339362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.339366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.339553 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.339942 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.339950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.339954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.340231 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:43.340377 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:43.340738 (+ 361us) server_negotiation.cc:299] Negotiation successful
0809 20:29:43.341022 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":46}
I20260809 20:29:43.341575 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.328070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34840)
0809 20:29:43.328227 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:43.328246 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:43.328337 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:43.328672 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:43.328677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:43.329137 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:43.329332 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:43.329343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.330404 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.330413 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:43.331381 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:43.331394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.332164 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.332170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:43.332359 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.333055 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:43.333072 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:43.335322 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:43.338661 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:43.338684 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:43.338690 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:43.339229 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:43.339672 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:43.339676 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:43.339680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:43.339839 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:43.340356 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:43.340361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:43.340496 (+ 135us) client_negotiation.cc:769] Sending connection context
0809 20:29:43.340664 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 20:29:43.341196 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":164}
I20260809 20:29:43.343302 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.327476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.12.78.61:44675)
0809 20:29:43.327652 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.327658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.327677 (+ 19us) server_negotiation.cc:407] Connection header received
0809 20:29:43.327795 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.327800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.327875 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.327995 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.329645 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.330329 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.331509 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.331912 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.332045 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.336362 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:43.336368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:43.336371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:43.336402 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:43.339965 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.340693 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.340701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.340704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.340859 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.341299 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.341303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.341305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.342166 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:43.342523 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:43.342714 (+ 191us) server_negotiation.cc:299] Negotiation successful
0809 20:29:43.342916 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 20:29:43.343241 4275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } attempt: 4
I20260809 20:29:43.347512 4440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f: Bootstrap starting.
I20260809 20:29:43.349488 12600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 20:29:43.351473 4440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:43.352736 4440 log.cc:824] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f: Log is configured to *not* fsync() on all Append() calls
Aug 09 20:29:43 dist-test-slave-rnrw krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307382, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.62@KRBTEST.COM for kudu/127.12.78.60@KRBTEST.COM
I20260809 20:29:43.356647 4440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f: No bootstrap required, opened a new log
I20260809 20:29:43.353001 4247 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "83f95430bd464b878c3d342e82a81efd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38674
I20260809 20:29:43.359551 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.343899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34856)
0809 20:29:43.344141 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:43.344163 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:43.344270 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:43.344885 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:43.344892 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:43.345297 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:43.345477 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:43.345486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.346976 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.346990 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:43.348661 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:43.348678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.349731 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.349737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:43.349936 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.350579 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:43.350606 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:43.353748 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:43.357028 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:43.357033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:43.357037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:43.357419 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:43.357809 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:43.357815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:43.357818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:43.357970 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:43.358547 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:43.358553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:43.358695 (+ 142us) client_negotiation.cc:769] Sending connection context
0809 20:29:43.358839 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 20:29:43.359158 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260809 20:29:43.359566 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.344323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.12.78.60:38075)
0809 20:29:43.344581 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.344587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.344614 (+ 27us) server_negotiation.cc:407] Connection header received
0809 20:29:43.344658 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.344662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.344718 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.344844 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.345634 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.346806 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.348811 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.349526 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.349705 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.353965 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:43.353971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:43.353974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:43.354006 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:43.356899 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.357538 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.357546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.357550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.357693 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.358080 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.358086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.358090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.358384 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:43.358576 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:43.358958 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 20:29:43.359171 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 20:29:43.364017 4275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd: Bootstrap starting.
I20260809 20:29:43.366779 4440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } }
I20260809 20:29:43.367147 4440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ad88be2e314663a55e07359393706f, State: Initialized, Role: FOLLOWER
I20260809 20:29:43.367662 4440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } }
I20260809 20:29:43.367970 4275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:43.368697 4448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } } }
I20260809 20:29:43.369107 4448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:43.369351 4275 log.cc:824] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:43.369801 4440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:43.372771 4275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd: No bootstrap required, opened a new log
W20260809 20:29:43.379518 4460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 20:29:43.379690 4460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 20:29:43.380421 4275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } }
I20260809 20:29:43.380869 4275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f95430bd464b878c3d342e82a81efd, State: Initialized, Role: FOLLOWER
I20260809 20:29:43.381369 4275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } }
I20260809 20:29:43.388306 4462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } } }
I20260809 20:29:43.388787 4462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:43.390509 4275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:43.390654 4247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 20:29:43.390831 4247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:43.377769 4365 master_runner.cc:186] Error getting master registration for 127.12.78.62:43109: OK, instance_id { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" instance_seqno: 1786307382198750 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 20:29:43.391499 4365 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 20:29:43.393584 4247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } }
I20260809 20:29:43.397123 4247 leader_election.cc:288] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [CANDIDATE]: Term 1 election: Requested vote from peers ca1060892656480ba8e5fe548de1a4d6 (127.12.78.61:44675), 51ad88be2e314663a55e07359393706f (127.12.78.60:38075)
I20260809 20:29:43.397652 4411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83f95430bd464b878c3d342e82a81efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51ad88be2e314663a55e07359393706f"
I20260809 20:29:43.398183 4411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:43.401650 4411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f95430bd464b878c3d342e82a81efd in term 1.
I20260809 20:29:43.402226 4220 leader_election.cc:302] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ad88be2e314663a55e07359393706f, 83f95430bd464b878c3d342e82a81efd; no voters:
W20260809 20:29:43.403852 4473 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 20:29:43.404057 4473 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 20:29:43.404937 4462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 20:29:43.406396 4200 master_runner.cc:186] Error getting master registration for 127.12.78.61:44675: OK, instance_id { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" instance_seqno: 1786307382730022 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 20:29:43.423106 4200 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 20:29:43.437307 4224 tablet.cc:2369] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:29:43.457768 4357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } attempt: 5
I20260809 20:29:43.471283 4462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [term 1 LEADER]: Becoming Leader. State: Replica: 83f95430bd464b878c3d342e82a81efd, State: Running, Role: LEADER
I20260809 20:29:43.473176 4462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } }
Aug 09 20:29:43 dist-test-slave-rnrw krb5kdc[4187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786307382, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.78.61@KRBTEST.COM for kudu/127.12.78.60@KRBTEST.COM
I20260809 20:29:43.503871 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.434219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.12.78.61:44675)
0809 20:29:43.434416 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.434423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.452418 (+ 17995us) server_negotiation.cc:407] Connection header received
0809 20:29:43.455995 (+ 3577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.456002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.456095 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.456172 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.464441 (+ 8269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.465603 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.473871 (+ 8268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.474569 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.474703 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.490053 (+ 15350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:43.490061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:43.490066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:43.490116 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:43.497633 (+ 7517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.498473 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.498479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.498482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.498666 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.499378 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.499384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.499388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.499794 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:43.500039 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:43.503215 (+ 3176us) server_negotiation.cc:299] Negotiation successful
0809 20:29:43.503487 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 20:29:43.505280 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.427519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.12.78.62:43109)
0809 20:29:43.427759 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.427766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.448942 (+ 21176us) server_negotiation.cc:407] Connection header received
0809 20:29:43.452789 (+ 3847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.452796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.452908 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.452991 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.460848 (+ 7857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.462171 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.471495 (+ 9324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.472328 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.472451 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.484872 (+ 12421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:43.484880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:43.484885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:43.484951 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:43.488919 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.496507 (+ 7588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.496514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.496518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.496747 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.500605 (+ 3858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.500625 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.500628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.501106 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:43.501371 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:43.504485 (+ 3114us) server_negotiation.cc:299] Negotiation successful
0809 20:29:43.504792 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260809 20:29:43.508322 4387 tablet.cc:2369] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 20:29:43.508891 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.449610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.12.78.60:38075)
0809 20:29:43.449801 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.449807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.455704 (+ 5897us) server_negotiation.cc:407] Connection header received
0809 20:29:43.456839 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.456845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.456930 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.457004 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.467553 (+ 10549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.468682 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.475693 (+ 7011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.476391 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.476502 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.491143 (+ 14641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:43.491152 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:43.491158 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:43.491203 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:43.495111 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.497258 (+ 2147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.497264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.497268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.497426 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.502008 (+ 4582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.502014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.502018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.502376 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:43.502643 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:43.508189 (+ 5546us) server_negotiation.cc:299] Negotiation successful
0809 20:29:43.508471 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 20:29:43.513150 4480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83f95430bd464b878c3d342e82a81efd. Latest consensus state: current_term: 1 leader_uuid: "83f95430bd464b878c3d342e82a81efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } } }
I20260809 20:29:43.513690 4480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:43.515036 4488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 20:29:43.517922 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.458483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.78.60:38075 (local address 127.0.0.1:34876)
0809 20:29:43.458917 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0809 20:29:43.458942 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 20:29:43.459059 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 20:29:43.479631 (+ 20572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 20:29:43.479638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 20:29:43.480266 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0809 20:29:43.480556 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:43.480577 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.486347 (+ 5770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.486355 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:43.488457 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 20:29:43.488481 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.489863 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.489872 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 20:29:43.490202 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.492359 (+ 2157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 20:29:43.492387 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 20:29:43.495907 (+ 3520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 20:29:43.514479 (+ 18572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:43.514485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:43.514490 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:43.515128 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:43.515648 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 20:29:43.515653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 20:29:43.515658 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 20:29:43.515856 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 20:29:43.516610 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 20:29:43.516645 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 20:29:43.516913 (+ 268us) client_negotiation.cc:769] Sending connection context
0809 20:29:43.517123 (+ 210us) client_negotiation.cc:240] Negotiation successful
0809 20:29:43.517519 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281}
I20260809 20:29:43.519253 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:43.459417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34876 (local address 127.12.78.60:38075)
0809 20:29:43.479069 (+ 19652us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.479078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.479110 (+ 32us) server_negotiation.cc:407] Connection header received
0809 20:29:43.479176 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.479183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.479287 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.480749 (+ 1462us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.481066 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.486015 (+ 4949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.488754 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:43.489647 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:43.497850 (+ 8203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:43.498174 (+ 324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 20:29:43.498184 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 20:29:43.498191 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 20:29:43.498232 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 20:29:43.514293 (+ 16061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.515311 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.515317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.515321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.515488 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 20:29:43.515984 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 20:29:43.515989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 20:29:43.515993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 20:29:43.516359 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 20:29:43.516495 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:43.518539 (+ 2044us) server_negotiation.cc:299] Negotiation successful
0809 20:29:43.518832 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19466,"thread_start_us":19967,"threads_started":1}
I20260809 20:29:43.520921 4488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 20:29:43.525480 4357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6: Bootstrap starting.
I20260809 20:29:43.529641 4357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6: Neither blocks nor log segments found. Creating new log.
I20260809 20:29:43.530967 4357 log.cc:824] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6: Log is configured to *not* fsync() on all Append() calls
I20260809 20:29:43.538154 4411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f [term 1 FOLLOWER]: Refusing update from remote peer 83f95430bd464b878c3d342e82a81efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:29:43.539202 4462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:29:43.544898 4357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6: No bootstrap required, opened a new log
I20260809 20:29:43.574087 4357 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } }
I20260809 20:29:43.574369 4448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83f95430bd464b878c3d342e82a81efd. Latest consensus state: current_term: 1 leader_uuid: "83f95430bd464b878c3d342e82a81efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } } }
I20260809 20:29:43.574688 4448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:43.574815 4357 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1060892656480ba8e5fe548de1a4d6, State: Initialized, Role: FOLLOWER
I20260809 20:29:43.576108 4357 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } }
I20260809 20:29:43.580261 4493 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } } }
I20260809 20:29:43.581002 4493 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:43.581102 4489 mvcc.cc:205] Tried to move back new op lower bound from 7316715042958753792 to 7316715042717184000. Current Snapshot: MvccSnapshot[applied={T|T < 7316715042958753792}]
I20260809 20:29:43.582752 4491 mvcc.cc:205] Tried to move back new op lower bound from 7316715042958753792 to 7316715042717184000. Current Snapshot: MvccSnapshot[applied={T|T < 7316715042958753792}]
I20260809 20:29:43.590797 4357 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 20:29:43.590850 4330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83f95430bd464b878c3d342e82a81efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ca1060892656480ba8e5fe548de1a4d6"
I20260809 20:29:43.591287 4329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 20:29:43.596306 4448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83f95430bd464b878c3d342e82a81efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } } }
I20260809 20:29:43.597452 4448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:43.597944 4329 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6 [term 1 FOLLOWER]: Refusing update from remote peer 83f95430bd464b878c3d342e82a81efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 20:29:43.599411 4480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83f95430bd464b878c3d342e82a81efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } } }
I20260809 20:29:43.599713 4480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:43.600831 4488 catalog_manager.cc:1347] Generated new cluster ID: af5a3bab6b4e40a793e192fce2d77536
I20260809 20:29:43.600935 4488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 20:29:43.601609 4330 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f95430bd464b878c3d342e82a81efd in term 1.
I20260809 20:29:43.613477 4462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 20:29:43.627189 4511 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 20:29:43.627288 4493 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83f95430bd464b878c3d342e82a81efd. Latest consensus state: current_term: 1 leader_uuid: "83f95430bd464b878c3d342e82a81efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } } }
I20260809 20:29:43.627595 4493 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:43.628170 4509 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83f95430bd464b878c3d342e82a81efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } } }
I20260809 20:29:43.628481 4509 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 20:29:43.636479 4480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83f95430bd464b878c3d342e82a81efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f95430bd464b878c3d342e82a81efd" member_type: VOTER last_known_addr { host: "127.12.78.62" port: 43109 } } peers { permanent_uuid: "ca1060892656480ba8e5fe548de1a4d6" member_type: VOTER last_known_addr { host: "127.12.78.61" port: 44675 } } peers { permanent_uuid: "51ad88be2e314663a55e07359393706f" member_type: VOTER last_known_addr { host: "127.12.78.60" port: 38075 } } }
I20260809 20:29:43.636782 4480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd [sys.catalog]: This master's current role is: LEADER
I20260809 20:29:43.642299 4488 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 20:29:43.644217 4488 catalog_manager.cc:1504] Loading token signing keys...
I20260809 20:29:43.673246 4488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83f95430bd464b878c3d342e82a81efd: Generated new TSK 0
I20260809 20:29:43.674465 4488 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 20:29:43.783339 4445 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:43.768751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.12.78.62:43109)
0809 20:29:43.768960 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.768969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.781134 (+ 12165us) server_negotiation.cc:407] Connection header received
0809 20:29:43.781671 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.781678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.781778 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.781855 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.783071 (+ 1216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38694: BlockingRecv error: recv got EOF from 127.0.0.1:38694 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260809 20:29:43.784910 4362 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:43.768874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.12.78.61:44675)
0809 20:29:43.769084 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.769091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.781237 (+ 12146us) server_negotiation.cc:407] Connection header received
0809 20:29:43.782685 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.782691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.782782 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.782853 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.784686 (+ 1833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44742: BlockingRecv error: recv got EOF from 127.0.0.1:44742 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 20:29:43.785715 4486 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:43.781796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.12.78.60:38075)
0809 20:29:43.781960 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.781966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.784151 (+ 2185us) server_negotiation.cc:407] Connection header received
0809 20:29:43.784379 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.784384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.784484 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.784574 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.785486 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34890: BlockingRecv error: recv got EOF from 127.0.0.1:34890 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 20:29:43.792387 4362 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:43.790390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44744 (local address 127.12.78.61:44675)
0809 20:29:43.790557 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.790562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.791052 (+ 490us) server_negotiation.cc:407] Connection header received
0809 20:29:43.791226 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.791231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.791343 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.791448 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.792156 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44744: BlockingRecv error: recv got EOF from 127.0.0.1:44744 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260809 20:29:43.798590 4362 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:43.796644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44754 (local address 127.12.78.61:44675)
0809 20:29:43.796798 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.796806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.797258 (+ 452us) server_negotiation.cc:407] Connection header received
0809 20:29:43.797426 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.797431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.797519 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.797632 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.798398 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44754: BlockingRecv error: recv got EOF from 127.0.0.1:44754 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260809 20:29:43.805859 4362 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:43.803661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.12.78.61:44675)
0809 20:29:43.803826 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.803832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.804314 (+ 482us) server_negotiation.cc:407] Connection header received
0809 20:29:43.804575 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.804580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.804680 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.804788 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.805607 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44768: BlockingRecv error: recv got EOF from 127.0.0.1:44768 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 20:29:43.816264 4362 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:43.813959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.12.78.61:44675)
0809 20:29:43.814168 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.814174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.814688 (+ 514us) server_negotiation.cc:407] Connection header received
0809 20:29:43.814887 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.814899 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.815002 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.815123 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.816033 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44782: BlockingRecv error: recv got EOF from 127.0.0.1:44782 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260809 20:29:43.830078 4362 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:43.827696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44790 (local address 127.12.78.61:44675)
0809 20:29:43.827883 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.827889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.828467 (+ 578us) server_negotiation.cc:407] Connection header received
0809 20:29:43.828681 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.828687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.828793 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.828918 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.829814 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44790: BlockingRecv error: recv got EOF from 127.0.0.1:44790 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 20:29:43.852865 4362 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:43.850107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.12.78.61:44675)
0809 20:29:43.850309 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.850316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.850906 (+ 590us) server_negotiation.cc:407] Connection header received
0809 20:29:43.851194 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.851202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.851312 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.851396 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.852622 (+ 1226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44800: BlockingRecv error: recv got EOF from 127.0.0.1:44800 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260809 20:29:43.893443 4362 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:43.890781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44812 (local address 127.12.78.61:44675)
0809 20:29:43.890997 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.891005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.891596 (+ 591us) server_negotiation.cc:407] Connection header received
0809 20:29:43.891850 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.891858 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.891962 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.892099 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.893163 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44812: BlockingRecv error: recv got EOF from 127.0.0.1:44812 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260809 20:29:43.963687 4362 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:43.961460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.12.78.61:44675)
0809 20:29:43.961704 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:43.961712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:43.962080 (+ 368us) server_negotiation.cc:407] Connection header received
0809 20:29:43.962298 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:43.962302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:43.962375 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:43.962462 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:43.963443 (+ 981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44826: BlockingRecv error: recv got EOF from 127.0.0.1:44826 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260809 20:29:44.097069 4362 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:44.095113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.12.78.61:44675)
0809 20:29:44.095301 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:44.095306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:44.095750 (+ 444us) server_negotiation.cc:407] Connection header received
0809 20:29:44.095892 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:44.095896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:44.096011 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:44.096100 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:44.096887 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44828: BlockingRecv error: recv got EOF from 127.0.0.1:44828 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260809 20:29:44.360129 4362 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 20:29:44.358300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.12.78.61:44675)
0809 20:29:44.358525 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:44.358531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:44.358792 (+ 261us) server_negotiation.cc:407] Connection header received
0809 20:29:44.358984 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:44.358988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:44.359061 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:44.359151 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 20:29:44.359894 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44838: BlockingRecv error: recv got EOF from 127.0.0.1:44838 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 20:29:44.381717 4460 catalog_manager.cc:1259] Loaded cluster ID: af5a3bab6b4e40a793e192fce2d77536
I20260809 20:29:44.381821 4460 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f: loading cluster ID for follower catalog manager: success
I20260809 20:29:44.384289 4460 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f: acquiring CA information for follower catalog manager: success
I20260809 20:29:44.386078 4460 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 51ad88be2e314663a55e07359393706f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 20:29:44.631582 4511 catalog_manager.cc:1259] Loaded cluster ID: af5a3bab6b4e40a793e192fce2d77536
I20260809 20:29:44.631760 4511 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6: loading cluster ID for follower catalog manager: success
I20260809 20:29:44.634344 4511 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6: acquiring CA information for follower catalog manager: success
I20260809 20:29:44.636188 4511 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ca1060892656480ba8e5fe548de1a4d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 20:29:44.884999 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:44.877714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44846 (local address 127.12.78.61:44675)
0809 20:29:44.878524 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:44.878531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:44.878553 (+ 22us) server_negotiation.cc:407] Connection header received
0809 20:29:44.878622 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:44.878626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:44.878777 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:44.878887 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:44.879435 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:44.880334 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:44.882372 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:44.882836 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:44.882933 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:44.883570 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:44.884179 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:44.884292 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:44.884473 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 20:29:44.884587 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":537,"threads_started":1}
I20260809 20:29:44.901443 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:44.891688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34896 (local address 127.12.78.60:38075)
0809 20:29:44.895982 (+ 4294us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:44.895987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:44.896005 (+ 18us) server_negotiation.cc:407] Connection header received
0809 20:29:44.896108 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:44.896120 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:44.896278 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:44.896469 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:44.896981 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:44.897718 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:44.899127 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:44.899540 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:44.899679 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:44.900192 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:44.900801 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:44.900968 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:44.901042 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 20:29:44.901139 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4184,"thread_start_us":4047,"threads_started":1}
I20260809 20:29:44.928697 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:44.907961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.12.78.61:44675)
0809 20:29:44.908151 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:44.908157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:44.921029 (+ 12872us) server_negotiation.cc:407] Connection header received
0809 20:29:44.921354 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:44.921360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:44.921464 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:44.921533 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:44.922580 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:44.923782 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:44.926222 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:44.926897 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:44.926996 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:44.927718 (+ 722us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:44.927858 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:44.927932 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:44.928176 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 20:29:44.928270 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 20:29:44.929556 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:44.908151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.12.78.62:43109)
0809 20:29:44.921418 (+ 13267us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:44.921425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:44.921444 (+ 19us) server_negotiation.cc:407] Connection header received
0809 20:29:44.921563 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:44.921568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:44.922002 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:44.922199 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:44.923186 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:44.924236 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:44.926457 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:44.927084 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:44.927207 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:44.927876 (+ 669us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:44.928347 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:44.928447 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:44.929176 (+ 729us) server_negotiation.cc:299] Negotiation successful
0809 20:29:44.929266 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13142,"thread_start_us":13651,"threads_started":1}
I20260809 20:29:44.934552 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 4200
I20260809 20:29:44.935542 4269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:29:44.939204 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 20:29:44.921396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34898 (local address 127.12.78.60:38075)
0809 20:29:44.921774 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0809 20:29:44.921780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 20:29:44.929084 (+ 7304us) server_negotiation.cc:407] Connection header received
0809 20:29:44.930332 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 20:29:44.930338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 20:29:44.930444 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 20:29:44.930516 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 20:29:44.931241 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:44.932505 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:44.934483 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 20:29:44.935142 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 20:29:44.935293 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 20:29:44.938432 (+ 3139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 20:29:44.938571 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 20:29:44.938664 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 20:29:44.938823 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 20:29:44.938908 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260809 20:29:45.050252 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 4200
W20260809 20:29:45.073628 4304 connection.cc:535] server connection from 127.0.0.1:44706 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 20:29:45.074077 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 4283
I20260809 20:29:45.075145 4351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:29:45.181402 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 4283
I20260809 20:29:45.203680 12600 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu pid 4365
I20260809 20:29:45.204913 4435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 20:29:45.320886 12600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktNpCrv/build/asan/bin/kudu with pid 4365
2026-08-09T20:29:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5049 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9637 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (383489 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST